TCM feb RG2 10/11/00 6:07 pm Page 1 Total Communication The monthly report on research, measurement and evaluation Reprinted from www.melcrum.com Communication audits can turn up disappointing news, but even lackluster results are a baseline for improvement. In this feature, Rodney Gray talks about recent data from Australia. by Rodney Gray The fine art of the audit As elsewhere, audits are increasingly popular in Australia, even though they don t always measure the most critical indicators of effectiveness. While the primary objective of measuring internal is to determine the extent it changes behavior or influences attitudes, the first step is to satisfy employees needs. Communication audits assess the extent to which employees needs are being met. Not only that, but good audits indicate employees assessment of a wide range of variables. By using factorial modelling, different aspects of can be correlated with overall satisfaction to reveal high leverage areas. This analysis reveals employees priorities. Communicators can then (a) concentrate improvements in those high leverage areas where there are both low scores and a high correlation with overall satisfaction; and (b) minimize efforts and expense where there s little impact overall. The who, what and how While I ve been conducting audits in Australia for many years, it s only been in the last few years that the research findings have been gathered in a single database to enable benchmarking. At present the database comprises the findings from 12 surveys from 10 Australian organizations. These include: one of the defence forces; an airline; a research organization; a government department; a retail bank; a merchant bank; a teles company; and three insurance companies. They range in size from a few hundred employees up to 16,000 and 22,000. Some are leaders in their field. Some of our surveys have been conducted internationally within Australian global organizations. These surveys, together with other anecdotal evidence, suggest that Australian findings differ little from those likely to be found in North American and European organizations. Australian business practices, after all closely model those in the US and UK. The questionnaires used had a minimum of about 35 items. They gathered opinions on five or six items on each of such internal topics as: upward, downward, horizontal, change, job, team, supervisor, manager and executive. The questionnaires used a five point Likert response scale, with all items worded positively. Each questionnaire had a write-in comments section. Also, our audits have usually included both qualitative and quantitative aspects, sometimes with one or two dozen focus groups conducted in various locations throughout
TCM feb RG2 10/11/00 6:07 pm Page 2 Australia. These give employees the opportunity to discuss generally and explain why questionnaire ratings are as they are. Presenting the findings How do employees rate? For simplicity, the database s positive median is the percentage of respondents who agreed or strongly agreed with the statements in the items making up each core-issue in each survey. The negative median is the percentage who disagreed or strongly disagreed with the statements in each core-issue. Those who did not respond either positively or negatively make up the balance. The medians are shown in the table. What s important to employees? A factorial modelling technique, similar to factor analysis, is used to sort responses statistically into core-issues. Each core-issue is then correlated with the questionnaire items making up the overall satisfaction core-issue. This is similar to conducting a regression analysis. The program uses respondents patterns of responses to indicate priorities. The typical correlations are shown in the table. What are the top priorities? The Performance-Prediction Matrix arrays the positive median scores (or performance ) for the various core-issues against the typical correlation ( predictive strength ) of each with overall The area of the matrix (see Figure 2, page five) where the top priorities lie is the bottom right hand corner: where those core issues with low scores and strong correlations appear. This graph shows that in Australia, generally speaking, there are six top priorities: senior management upward consultation and involvement change cross-functional company information. Naturally the findings do vary from one organization to another. But this pattern, with employees much more satisfied with local, workgroup than management, is very typical judging by the organizations in the database. What they told us Employees say that company fails them. The audits conducted here reveal fully 62% are not satisfied. A little more than a third of surveyed Australian employees (38%) give positive ratings of overall satisfaction, and almost a quarter (26%) give negative ratings. The remaining 36% could not decide, or are partly, or sometimes, both positive and negative. Yet, unquestionably, such low scores indicate that productivity is being hampered by poor internal. This is confirmed in focus groups with employees giving countless examples of how poor impacts their ability to do a good job. If these organizations are typical, and I believe they are, most organizations are underperforming because of poor employee. Even the best scores aren t good enough The table (below) shows the best scores recorded by various organizations in each of the coreissues. Given how low most of these scores are, we have not called them best practice. It s a depressing picture. Clearly there is considerable room for improvement overall, and in most areas surveyed. If 71% of surveyed employees (on average) are positive about job information, why not other aspects of? What are organizations doing to communicate job Figure 1: Typical correlations, best scores and medians for core issues Typical Best Correlation Score Medians (+%) +% -% Overall satisfaction 1.0 52 38 26 Extremely strong correlates Senior Management Team 0.7 54 38 26 Upward 0.7 64 41 31 Very strong correlates Change 0.65 53 36 29 Consultation and involvement 0.65 50 40 37 Strong correlates Cross-functional 0.55 45 30 36 Company information 0.55 58 42 24 Moderate correlates Strategic direction 0.5 86 51 16 Immediate manager or supervisor 0.5 67 61 17 Team meetings & 0.5 77 59 19 Job information 0.5 86 71 13 Performance feedback 0.5 63 53 24 Not very significant Company publication 0.3 56 48 14 Senior executive roadshows 0.3 60 57 8 Results presentations 0.3 55 52 19 E-mail messaging 0.3 55 44 29 Intranet / Lotus Notes 0.3 68 56 20 t Rodney Gray is principal at Employee Communication & Surveys in Sydney, Australia. Special thanks go to Geoff Alford of Geoff Alford Research Services, Melbourne, for all the computing for these audits, development and maintenance of the database, and production of tables and graphs. E-mail: rodneygray1 @compuserve.com 2
TCM feb RG2 10/11/00 6:07 pm Page 3 Lower level employees, especially those in blue collar roles, often want most from their immediate supervisor. However, perceptions of immediate supervisors are much more positive. If you plot the gap between employees perceptions of existing and what s wanted from various sources, the gap is almost always greatest (and usually by far) for the CEO. In one public sector agency there was a wide range of consultative mechanisms by way of crossfunctional project teams. Senior executives couldn t understand why employees were so negative about consultation. But employees wished to be involved in decision-making about things to do with their jobs and local work environment. t information that they re not doing for, say, cross-functional or change? The key findings In order to review what Australian employees think of with them in more detail, the median scores for each of a number of the main core-issue findings are discussed in five groups in the order they typically correlate with overall (The typical correlations shown below and in the table are the averages of the correlations in the database.) 1. Extremely strong correlations with overall satisfaction Senior management? Who are they? Employees often think senior managers do a pretty poor job of communicating with them. On average only 38% are positive, and 26% are negative. A poor picture indeed. In Australia, employees perceptions of senior management is statistically the factor which correlates most strongly with overall On average, senior management correlates 0.7 with overall Recorded correlations range from 0.45 to 0.8. No-one listens to us Ratings of upward in Australia are also mediocre. Only 41% of employees surveyed believe their organizations provide adequate opportunities for upward, and around a third say they definitely do not. Upward generally, especially listening by executives, is very important to employees and usually correlates extremely strongly with overall satisfaction (on average about 0.7; range 0.65-0.75). Overwhelmingly, our employees tell us that managers and executives must find ways to listen better. This finding is closely related to the next two core-issues: change and consultation and involvement. 2. Very strong correlates with overall satisfaction What s really going on around here? Australian employees perceptions of change are generally not favourable. As with a number of other factors, only about a third of employees have positive perceptions, and almost a third give negative scores. In some organizations undertaking major change initiatives, this aspect of internal vies with senior management for the highest correlation with overall Correlations with overall satisfaction of 0.6-0.7 are typical, even in those organizations not undergoing major change. Ideally, employees want to be consulted and involved in change efforts. But, failing this, they insist on getting good about the changes that are happening or planned in their organizations. Some leading organizations do get this right. They make a very active effort to include employees in the planning and implementation of change initiatives. I want to be involved In today s involve me society, employees want to be consulted, involved and valued for their contributions in matters to do with their work and immediate work environment. This especially applies in the case of the increasing numbers of knowledge workers. Consultation & involvement median scores are 40% positive and 37% negative. These show that most organizations are not yet involving employees in decision making to the extent they want. In recent years, this aspect of upward has become critical in most Australian organizations. A very strong correlation with overall satisfaction of around 0.6 is typical (range 0.5-0.7). 3. Strong correlates with overall satisfaction We re more like a dozen different companies The medians for cross-functional are only 30% positive and 36% negative. Even the best score recorded in nine organizations is only 45% positive. These are the worst scores in the database. And cross-functional correlates as highly as 0.7 (typically 0.55; range 0.4-0.7) with overall Despite the efforts of the total quality management (TQM) movement over the last couple of decades, cross-functional or crossdepartment is usually extremely poor in Australia. Even in our best organizations, those which have made great strides to improve downward and upward, from one function or department to another is often lacking or of poor quality. 3
TCM feb RG2 10/11/00 6:07 pm Page 4 It s just head office propaganda Despite huge efforts and considerable expense for Intranets, ad hoc e-mail and fax news services, senior executive roadshows, results presentations, as well as award-winning company newsletters and magazines, less that half (42%) of surveyed Australian employees are satisfied with organizational or company information. And a quarter are not satisfied. Company information generally correlates around 0.55 (range 0.4-0.65) with overall 4. Moderate correlates with overall satisfaction Where are we heading? Communication of where the organization is going can be significant to employees, particularly in times of change. The median score indicates that 51% of employees are satisfied with of strategic direction to them. And 16% are not. Strategic direction correlates moderately (range 0.4-0.55) with overall My boss is not the problem In Australia, 61% of surveyed employees are positive about with their immediate boss. This is one of the highest scores in the database. But clearly there is room for improvement. While only 17% were negative, another 22% (the neutrals ) were not prepared to rate their manager or supervisor positively. The immediate manager or supervisor correlates moderately (around 0.4 to 0.6) with overall Communication in my team is okay Ratings of team meetings and (and the typical correlation with overall satisfaction) are much the same as those for immediate managers or supervisors. I know what I have to do... The picture with of basic job information is even better, with 71% of employees being happy with this. Only 13% do not feel job is satisfactory. Australian organizations have made a major effort in this area over the past five years or so. Job information correlates moderately (0.4 to 0.6) with overall...but no-one tells me how I m doing Unfortunately, the excellent job picture is not matched by performance feedback to employees. The positive median for performance feedback is 53% with a negative median of 24%. Employees often report in focus groups that they assume no news is good news. Performance feedback correlates moderately (0.4-0.55) with overall 5. Not very significant to overall satisfaction In Australia (and likely in other developed countries) employees in focus groups can discuss for an hour or more without ever mentioning corporate publications or intranets, although email generally gets a mention. Consider the data: Print publications are still popular with Australian employees, with 48% satisfied with their organisation s publication, and only 14% dissatisfied. But publications contribute little to overall Recorded correlations range from 0.3 to 0.45. The median level of satisfaction with senior executive presentations and roadshows is 57% positive and only 8% negative. Clearly most employees like executive roadshows, but they contribute very little to overall satisfaction with correlations from 0.3-0.4. The findings for annual or half-yearly senior executive results presentations are much the same as for executive roadshows. More than half the employees like these, but they make little impact on overall Email is viewed favourably by 44% of employees, with 29% not satisfied. The correlation with overall satisfaction is relatively low, typically around 0.2-0.4. As to intranets, employees are 56% positive and 20% negative about intranets and/or Lotus Notes systems. However, as with email, these tools are not very significant to overall So what does it all mean? One clear message emerging from these surveys: tools for (or information flow) should not be high priorities for communicators. Communicators should be concentrating their efforts on helping executives and managers improve their, especially listening to employees, and consulting and involving them in the changes needed in the workplace. The scores would have been higher but for some organizations having multiple incompatible e-mail systems as a result of mergers. A more difficult problem to overcome is over-using e-mail at the expense of interpersonal, even with people in offices or cubicles close by. You may not like what you find out when you conduct a audit or other research of employees. However, if you can find out exactly how to improve and take appropriate action, you can make a big improvement to organizational productivity. 4
TCM feb RG2 10/11/00 6:07 pm Page 5 HOW... to prepare a drive analysis matrix Suppose you re faced with an array of issues that you ve identified as possible drivers affecting a problem in your company. Attention to any one of them might help, but how much attention can you afford to give? Naturally, since your resources are limited, you ll want to focus on the drivers that will have the most impact on the results you want. How do you decide where to start? Developing a driver analysis matrix can help. It s time for the pay off In this issue s Feature article (see pages 1-4), measurement and research consultant Rodney Gray outlines a s audit and offers two methods for presenting and organizing research data. The first, a correlations table, is reproduced on page 2. A second, the Performance Prediction Matrix, appears below. The matrix demonstrates a useful technique for arranging data via driver analysis that makes it easier to identify the issues that are most likely to give the biggest payoff. Figure 2: performance-prediction matrix: correlates of internal satisfaction Key issues in vision The matrix shown here (right) ranks core issues according to their percentage of agreed/strongly agreed respondents along the left-hand side in ascending order. The horizontal axis of the chart plots the predictive correlation for each core issue. By arraying core issues on the chart according to the points of intersection between the two axes, some startling visual information emerges. This type of matrix quickly highlights core issues that while scoring low on the left-hand side have a strong predictive value. In other words, a dollar or a franc spent addressing these issues will probably yield higher rewards and a greater impact on satisfaction than efforts devoted to issues falling elsewhere on the matrix. In the (strictly theoretical) Figure 3 (right), four issues identified as drivers of effective are scattered across four quadrants of a matrix created from two axes valued from Figure 3: Priority predictor matrix zero to 100. Issues falling above the horizontal matrix have scored reasonably well for specific satisfaction on a (fictitious) survey, while those falling below have fared poorly. For example, e-mail access is an issue that while low on the satisfaction index also registers low for its predictive value as a driver of overall effectiveness. Through the eye of a needle Applying resources to improve e-mail access could result in raising specific satisfaction levels, with little impact on the overall outcome. On the other hand, attention paid to the bottomright quadrant in Figure 3 will most likely move the needle upward vertically and to the right horizontally. Resources expended to improve credibility would have a much higher probability for cost effectiveness. This is why envisioning such a matrix before collecting your data is useful. It helps to shape the questions best calculated to produce the kind of data you need to develop the matrix in the first place. And that means questions that not only ask about specific levels of satisfaction, but also aim to form an understanding about impact on your overall goal. A matrix coordinating issues with value can provide powerful arguments for your recommendations. It can be tempting to grab at the lowhanging fruit, especially when the temptation is reinforced by budget constraints. Yet it s often the harder issues that will reward your efforts the most. Thanks to Geoff Alford for his help with the tables on this page