An Organizational Culture Consistency Meter in Evaluation of Organizational Learning

Transcription

1 An Organizational Culture Consistency Meter in Evaluation of Organizational COLIN A. SHARP JOHN STANWICK AND JOANNE BAULDERSTONE Flinders Institute of Public Policy and Management Flinders University, GPO Box 2100, Adelaide, Australia, 5001 ABSTRACT In the management literature, theories of organizational culture and organizational learning are frequently assumed to be valid models for management of change. However, adequate development of assessment devices for longitudinal organizational research in these areas has been limited, undermining the validity and applicability of such theory. To address this concern, this paper introduces a Computer Aided Text Analysis (CATA) based method of assessing the consistency of espoused culture in corporate vision or strategic planning documents, enabling comparison with the culture perceived by staff of the organization. The present paper reports an exploratory application of CATA to the strategic documents of four Australian public sector organizations. The CATA data was used to give an indication as to the dominant culture type espoused in a corporate document. Comparison of the CATA data, with results of staff surveys of the organizational culture showed some potential gaps between the espoused and the actual perceived culture of the organizations. The prospect exists that this approach to identifying culture gaps can assist managers as an organizational culture consistency meter, and that this triangulation of methods may provide researchers a method of measurement of changes in organizational culture as an indicator of organizational learning. INTRODUCTION According to the Secretary of the central agency of the Australian Government responsible for the Financial Management Improvement Program (FMIP), "One of the most exciting developments in recent years has been the growing emergence of an evaluative culture." (Sedgewick, 1994, p. 22) For more than twelve years, the Commonwealth Department of Finance presided over a whole of government reform process, including establishing a system of program evaluation and accountability for outcomes. Sedgewick (1994; 1995) has argued that the high incidence of program evaluation and "benchmarking of international best practice" are indications of a culture of continuous improvement. An "evaluative culture", was described as an attitude, a body of technology and "incentives", which lead to a questioning by managers and ministers of the effectiveness, efficiency, and appropriateness of government programs (Sedgewick, 1994, p. 20).

2 Of course these comments were made more than a year prior to the overwhelming electoral victory of the Liberal (conservative) party of Australia, which set about radically 'down-sizing' the public sector. Nevertheless, there is a proposition (put forward by senior central agency bureaucrats) that in Australia over the last ten to thirteen years, the Australian Public Service (Commonwealth government administration) has been trying to create a "culture of evaluation" as part of a general programme of continuous performance improvement and organizational learning. Questions are asked here about how public sector organisations can evaluate organizational learning. How would they be able to demonstrate the claimed cultural change? How would the effectiveness of a whole of Government program of organizational cultural change be evaluated? The intention is not to review the various evaluation studies conducted through the Australian Government's Evaluation Strategy, nor its reviews of its own progress in achieving reforms. Rather, it is proposed that we need an "Organisational Culture Consistency Meter" (Sharp, 1993; 1996; Sharp, Stanwick & Baulderstone, 1997). This analytical system would draw on learning theory to provide an index of organizational learning which identifies organizational culture gaps between espoused values and values in use, as an indicator of the probability of organizational learning (see Sharp, 1994; 1996). With such a tool we can better understand the measurement of change and the refinements of methodology of evaluation required to be able to assess the effectiveness of organisations attempting goals of this kind. In attempting to operationalize the theory and measures of such proposed changes, a new method of organisation culture gap analysis has been developed (Sharp, 1993; 1996; Sharp, Stanwick & Baulderstone, 1997). It uses Computer Aided Text Analysis (CATA) of the key strategic documents of the organisation to identify the "espoused culture" of the organisation, which can be compared with a survey of the organisation's staff, based on the Competing Values Framework (CVF), but modified specifically for use in the public sector, as an indicator of "values in use". This paper will report some preliminary research findings which suggest that the culture gap technique has some potential to identify indications of the potential direction of organizational learning or organisation culture change in the public sector. METHOD The standardised instrument chosen to operationalise these measurements was the Competing Values Framework (CVF) survey (Quinn & Rohrbaugh, 1983). It "measures" values in two dimensions, viz: flexibility - rigidity, and internal - external focus. The intersection of these two dimensions result in four cultures types. These are: the human relations model: (flexibility/internal); the open systems model: (flexibility/external); the rational goal model: (rigidity/external); the internal process model: (rigidity/internal). The CVF provides a body of literature, and it has already been used in Australia by Stanwick (1991; 1993). It can also be used in content analysis of documents of organizations, by drawing on the work of Jewell (see Jewell, 1993; Jewell & Linard, 1992). He has shown through the development and application of PLANSPEC, that the information management capabilities of Lotus (R) Agenda are useful in analysing the semiotics of organizations as expressed in the corporate planning and reporting documents.

3 Using a new version of PLANSPEC, that we have developed at Flinders University, called CULTURESPEC, the items from an organisation's strategic plan and corporate reports appear as cells in a spreadsheet-type analysis of information. These can then be compared with regard to their relationships to the categories in the Competing Values Framework, to enable subsequent interpretation of that information in terms of the staff values survey. In addition to its application to strategic plans and corporate reports, CULTURESPEC was applied to semi-structured interview data obtained from one of the organizations in the sample. Four public sector organizations (called organizations I to IV) were selected for initial comparison of computer assisted text analysis (CATA) and surveys of staff values (using the Competing Values Survey, and one being developed by Stanwick (1993; 1994)). These organizations were chosen because they differ in terms of their client focus (resources management versus services to children) and sector (including federal and state). Kendall's W was used as the test statistic it provides a non-parametric test of the hypothesis that several related samples are from the same population. It is highly suitable for this type of qualitative data and comparative analysis being a test of agreement among raters' rankings of various items by the same criteria. RESULTS Table 1. Comparison of CATA Ranks and CVF Survey Ranks for Organizations II and III Culture Type Human Relations Open Systems Internal Process Rational Goal Doc. 1 Org. I Doc. 2 Org. I Doc. 3 Org. II Org. II CATA Survey CATA Survey CATA Survey Interviews CATA

4 Table 2. Comparison of CATA Ranks and OCS Actual And Desired Culture Survey Ranks for Organizations III and IV Culture Type Human Relations Open Systems Internal Process Rational Goal Doc. 4 Org. III Doc. 5 Org. IV Doc. 6 Org. IV CATA Actual Desired CATA Actual Desired CATA Actual Desired Comparison of survey (and interview) data with the CATA of the strategic documents indicate that there are differences in the "espoused" culture as evidenced by corporate documents, and the perceived "enacted'" culture as indicated by staff survey. Indeed, there were no statistically significant concordances between the CATA and survey results, or the CATA of strategic documents and the CATA of the semi-structured interview in Organization III. DISCUSSION Some of the data of Sharp Stanwick and Baulderstone (1997) bear on the question of an evaluative culture in the Australian public service. If this proposition (Sedgewick, 1994) was correct then it would be expected that the CVF culture type most dominant would be the Rational Goal Model explained by Quinn and Rorhbaugh (1983) as results-oriented, externally focused on customers and goals. This is very much akin to the prevailing paradigm of program evaluation in Australia (MAB-MIAC, 1993; Sedgewick, 1994). However, if the data from each of the CATA analyses are examined (see Tables 1 and 2) none of the results rank the "Rational Goal Model" as the predominant culture type. According to the staff survey data (see Tables 1 & 2) none of the organizations showed the "Rational Goal Model" as the dominant culture. The CATA of the interview data from Organization II also did not rank the "Rational Goal Model" as the dominant culture type. Even in Organizations III and IV, where staff surveys included perceptions of both Desired and Actual cultures, again there was no preference for the "Rational Goal Model". These analyses were conducted among both state and commonwealth organisations over the last three years, during which time Sedgewick (1994) made his assessment of the development of an evaluative culture. Even within the South Australian Government, between the years of the Bannon Labour Government employed a policy of promoting evaluation, and the early days of this current research were during this period. It is difficult to resolve the discrepancy between Sedgewick's assessment and the data obtained by Sharp et al (1997).

5 Indeed, it would appear that "Internal Process Model", which is akin to the more 'traditional' Public Service organization culture type (formal rules and procedures, predictability, internally focused) is the dominant culture type in some of the documents analysed, as well as survey and interview results. For example, the CATA of documents 1 and 4, the "actual culture" survey results for all the organizations except for Organization II, and the interview data for organization II all show the "Internal Process Model" as the dominant culture type. (However, a follow-up questionnaire in Organization II, which was done about 18 months after the original study, and not included in this analysis, indicated that the "Internal Process Model" was the dominant culture type.) LIMITATIONS AND FURTHER RESEARCH There are limitations and weaknesses in the both the CATA data and staff survey data as measures of organizational culture on their own. To begin with the data obtained contains data from only one Commonwealth Government organization, the rest being State Government organizations. Validity of the survey data was limited with typically poor response rates from the organizations surveyed (although in total approximately 600 surveys were returned and analysed). This may be a consequence of the turbulent times facing the public sector in Australia recently (e.g., see Sharp & Lewis, 1993; Sharp, 1996). Such low response rates do raise concerns about the degree of generality of these findings for the diagnosis of the culture of these specific organizations. However, the results of the surveys were accepted by the senior staff of the organizations concerned, indicating at least some "face validity". A further problem lay with the reliability of the CATA software. The choice of CATA software, Lotus (R) Agenda (Sanders, 1989), proved problematic because of its limitations of "user-friendliness" and unpredictability. This led to only a limited application of its potential to deliver quantitative data, compared to Jewell and Linard's (1992) expectations. Thus the researchers had to print out the computer's classification of each paragraph and count the number of times each culture type was identified, and then perform the data analyses on those numbers. Not only was this laborious, but it added the risk of inaccuracies in the data compilation. Furthermore, the development of CULTURESPEC was developed in Lotus (R) Agenda software, and there may be limitations in the transferability of the lexicon to other computer software. These and other methodological problems are symptomatic of the difficulties of the field of research and practice in organisational diagnosis and organizational learning in the public sector. The analysis provided above is only as persuasive as the data available to allow tests of the concept of an "evaluative culture". However, as mentioned, these are part of an ongoing longitudinal study investigating various public sector organizations. Surveying of public sector organisations from time to time can be used as a metaevaluation to determine the evaluative culture of public sector organizations. Given the many fundamental changes which have taken place in public sector management over the past ten or more years, it would be important to examine culture change in public sector organizations as a form of organizational learning.

6 REFERENCES Jewell, R.E PLANSPEC - A decision support system for developing and testing plans. Unpublished paper copyright 1992, to Lt-Col R.E. Jewell, Dept of Civil & Maritime Engineering, University College, Australian Defence Force Academy, Campbell, ACT 2600, Jewell, R.E & Linard, K.T "Evaluating corporate plans and the corporate planning process" Evaluation-Making it Work: Proceedings of the International Conference of the Australasian Evaluation Society, Melbourne, July, vol. 2, paper 59. MAB-MIAC 1993 The Australian Public Service Reformed: An evaluation of a decade of management reform. Report of the Task Force on Management Improvement of the Management Advisory Board & Management Improvement Advisory Committee, Canberra, AGPS. Quinn, R. E. and Rohrbaugh, J "A spatial model of organizational effectiveness criteria: Towards a competing values approach to organizational analysis". Management Science. vol., 29, no.3 pp Sedgewick, S, 1994, "Towards An Evaluative Culture: Making Performance Count", Evaluation News & Comment, Australasian Evaluation Society Conference, vol. 3, no. 2 (December), pp Sharp, C.A "An Organisational Culture Consistency Meter? - Diagnosing Organizational Culture Gaps from Annual Reports and Staff Survey". Unpublished paper presented to the ANZAM Conference, December, Deakin University, Geelong, Vic. Sharp, C, 1994, "Evaluation of Organisational : An Organisational Consistency Meter?", International Evaluation Conference of the Australasian Evaluation Society, Canberra, September. Sharp, C.A., 1996 "Evaluation of Organisational in the Public Sector: An Organisational Culture Consistency Meter?" Department of Management Lancaster University, September. Sharp, C.A., Stanwick, J. & Baulderstone, J "Computer Aided Text Analysis: An Organizational Culture Consistency Meter?" Paper submitted for publication (Flinders Institute of Public Policy and Management, January, 1997). Stanwick, J. 1993, "Instruments for the assessment of organisational cultures in changing public sector environments in Australia - A pilot study" Paper presented at the Australian and New Zealand Academy of Management (ANZAM) Conference, Deakin University, Australia, December. Stanwick, J. 1994, "The development of instruments for assessments of organisational cultures in changing Public sector environments in Australia". Paper presented at the Australian and New Zealand Academy of Management (ANZAM) Conference, Victoria University of Wellington, New Zealand, December.