Available online at www.sciencedirect.com Procedia Engineering 43 (2012 ) 331 337 International Symposium on Science and Engineering in China, 2012 (ISSSE-2012) The Development of a New Evaluation Index System Lei Wang a,b,*, Ruishan Sun a,b a Research Institute of Civil Aviation, Civil Aviation University of China, Tianjin, 300300, China b National Key Laboratory of Air Traffic Operational Technology, Tianjin, 300300, China Abstract culture has become an important issue in safety research and organizational safety management due to its latent implications on safety performance. The main aim of this research is to develop a reliable safety culture evaluation index system based on the Integrated Model (ISCM). Firstly two forms of safety culture evaluation index system were constructed and every index was described. Then the weights of index were determined by using Delphi method and the reliability of the index system was also examined based on data from survey. The results indicated that evaluation index system was effective and applicable and expected to be as the basis of final safety culture quantitative measurement tool. 2012 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of the Capital University of Economics and Business, China Academy of Science and Technology. Open access under CC BY-NC-ND license.. Keywords: culture,evaluation index system, ISCM 1. Introduction culture is a natural part of organizational culture and alludes to individual, job, and organizational features that affect and influence health & safety [1]. It was proposed out as a term by the International Nuclear Advisory Group (INSAG) of International Atomic Energy Agency (IAEA) in 1986 after their analysis on nuclear reactor accident at Chernobyl [2]. Generally within complex sociotechnical systems around 75% of all accidents and safety compromising incidents are attributed to human errors [3]. However, human failure of the front line operator, in many accidents, represents only a superficial cause. As a matter of fact, lots of accidents could be traced to organizational reasons [4-7]. The Swiss Cheese Model and organizational accident theory developed by James Reason indicated that the root reason of individual unsafe acts always could be found on organizational level, and organizational factors would affect human and local workplace factors through latent conditions pathways. culture is major part of these latent conditions and always determines the efficiency of safety management of high-risk-organizations. Standing on this perspective, it was generally accepted that organizational accident is the consequence of poor safety culture. In Fig.1 the evolution of a higher safety level is shown [8]. In the beginning, and still continuing, there were the improvements in technical safety by improved design and the introduction of the best available technology. Shortly after, human and safety management aspects start to play a major role. Currently the focus is on safety culture as we shall see later * Corresponding author. Tel.: +86-22-24092582; fax: +86-22-24957940. E-mail address: wanglei0564@hotmail.com 1877-7058 2012 Published by Elsevier Ltd. doi:10.1016/j.proeng.2012.08.057 Open access under CC BY-NC-ND license.
332 Lei Wang and Ruishan Sun / Procedia Engineering 43 ( 2012 ) 331 337 [8]. Managing safety has become increasingly more important in aviation and other high reliability, safety critical systems. culture will play a more critical role in improving safety as that the International Civil Aviation Organization pointed out An effective way to promote a safe operation is to ensure that an operator has developed a positive safety culture [9]. level Management focus Management System Attention to human factors Technical safety Year 1960 1970 1980 1990 2000 2010 Fig.1. Improvement of safety level and the contributing factors Lots of theories and models regarding with safety culture have been developed in such diverse domains as risk management (e.g., [10-11]), human factors [12], aviation (e.g., [13]) and so on. Reason [14], Cooper [1], Wiegmann [12] and Xu [15] both researched the definition of safety culture and identified the dimensions and components of safety culture. Though there have been numerous fruitful products in this filed, the tool of measuring safety culture in a quantitative way was rarely detected in previous research. Obviously safety culture measurement plays a key role in the safety improving process because it can evaluate the current state of a particular safety culture, as well as to determine whether interventions have been effective in achieving a desired cultural change. In the European Commission FP6 project called HILAS (Human Integration into the Lifecycle of Aviation Systems, AIP4-CT-2005-516181), one of our main tasks was to develop an applicable safety culture measurement tool for aviation organizations. Based on task requirements, we proposed out an Integrated Model (ISCM) and developed a complete safety culture measurement tool kit including evaluation index system, questionnaire and implementation program [16]. This tool kit has been applied into several airlines for safety culture survey in China and Europe. This paper was aiming to introduce the whole work of developing evaluation index system based on ISCM. The work presented in this paper provided support to the final safety culture evaluation tool. 2. Integrated safety culture model Based on literature review and complete analysis on structure and components of safety culture [1] [12] [14], a new definition of safety culture was proposed out from the perspective of sub-culture, that was described as culture is assembly of values, beliefs, attitudes, norms, organizational characteristic behaviors and environments which are concerned with safety, created and nourished in the long term process of organizational producing practice, and accepted by most of members in organization. Meanwhile, safety culture was separated as seven types of sub-culture which are safety Priority,, Flexible, Learning, Teamwork, Reporting and Just. The Integrated Model (ISCM) was developed based on the definition (Fig.2) [16]. In this model, the dimensions of safety culture are attributed into two levels, one is intrinsic latency level containing all dimensions of safety philosophy and the other one is extrinsic indication level containing all dimensions of safety environment and safety behavior. The eight dimensions of extrinsic indication level could be matched with the seven safety sub-cultures, and then the potential links between safety sub-culture and all safety culture dimensions are also demonstrated in this model. The components of each dimension will be discussed in the process of developing evaluation index system.
Lei Wang and Ruishan Sun / Procedia Engineering 43 ( 2012 ) 331 337 333 Extrinsic indication level Intrinsic latency level Priority Organizational Commitment Flexible Organization Regulation and Rule Operation environment philosophy values consciousness Learning Teamwork Information Exchange Education and Training behavior attitudes Reporting Management Just Rewards and Punishment Fig. 2. Integrated Model 3. The development of evaluation index system 3.1 Structure of index system A reliable evaluation index system is the basis of designing an effective safety culture questionnaire, therefore the index system should be complete and indices should be logical enough for accurate measurement on safety culture. Based on ISCM and expert discussion, two forms of safety culture evaluation index system were constructed and presented as Fig.3 and Fig.4, one of which was classified from the perspective of the extrinsic indication dimensions of safety culture, and another one was from the perspective of the sub-culture component. Guideline and Policy Objective and Responsibility Resource Allocation Organizational Commitment Priority Organization Setting and Function Organization Completeness and Applicability Executing and Fulfilling Regulation and Rule Flexible Learning Risk Management Staff Involvement Supervision Operational Skill Teamwork Skill Risk-resisting Capability Management Operation Values Consciousness Attitude Teamwork Quantity and Variety Quality and Effect Organizational Learning Education and Training Reporting Just Reporting System Information Submission Information Feedback Experience Sharing and Communication Justice and Equality Information Exchange Rewards and Punishment Fig. 3. Evaluation index system based on extrinsic indication dimension
334 Lei Wang and Ruishan Sun / Procedia Engineering 43 ( 2012 ) 331 337 Guideline and Policy Priority Objective and Responsibility Organization Setting and Function Resource Allocation Organizational Commitment Flexible Learning Completeness and Applicability Executing and Fulfilling Risk Management Staff Involvement Risk-resisting Capability Quantity and Variety Quality and Effect Operational Skill Organization Regulation and Rule Management Operation Values Consciousness Attitude Organizational Learning Teamwork Reporting Just Teamwork Skill Experience Sharing and Communication Reporting System Information Submission Information Feedback Supervision Justice and Equality Education and Training Information Exchange Rewards and Punishment Fig. 4. Evaluation index system based on safety sub-culture Obviously it is quite difficult to evaluate safety culture by measuring intrinsic dimensions like individual s philosophy or mental state. The ISCM has demonstrated that all intrinsic dimensions are infiltrated in extrinsic dimensions, so safety culture could be evaluated through measuring extrinsic dimensions. As seen that the third layer indices are the same between the two forms of index system, their differences just exist in position and sequence. These two forms make it possible to evaluate safety culture from the two different perspectives. 3.2 Description of index Each evaluation index is described as following. (1) Organizational Commitment Guideline and Policy: is to evaluate whether the organizational decision-making level developed specific safety guidelines and policies and behaved with a positive attitude of caring for safety. Objective and Responsibility: is to evaluate whether the organizational decision-making level mapped out safety objectives, and assigned corresponding safety responsibilities clearly to make sure that these objectives were reached. Resource Allocation: is to evaluate the status of allocating resources, human and for example equipment and tools, by the organizational decision-making level, and whether these resources can assist in ensuring workers safety. (2) Organization Organization Setting and Function: is to evaluate the status of establishing a safety management division, committee and manager in an organization, and whether they played a positive role in the process of accident prevention and dealing with accidents. (3) Regulation and Rule Completeness and Applicability: is to evaluate whether the safety regulations, rules and standards were completed, applied, and updated. Executing and Fulfilling: is to evaluate the status of executing regulations, rules and standards by an organization and each individual s abiding to those. (4) Management Risk Management: is to evaluate the status of developing a risk management procedure and implementing risk management in an organization. Supervision: is to evaluate whether an organization set up safety supervision mechanisms and the extent of strictness and justice (fairness) when implementing those. Staff Involvement: is to evaluate the extent of all staff involvement in safety decision-making and management activities. (5) Operation :
Lei Wang and Ruishan Sun / Procedia Engineering 43 ( 2012 ) 331 337 335 Operational Skill: is to evaluate an individual s operational skill and their attitude toward improving it. Teamwork Skill: is to evaluate the extent of an individual s trust in and cooperation with their colleagues. Risk-resisting Capability: is to evaluate an individual s capability and behaviour in identifying and controlling risk and dealing with emergencies. (6) Education and Training Quantity and Variety: is to evaluate the quantity and variety of deploying safety education and training. Quality and Effect: is to evaluate the quality and effectiveness of safety education and training. Organizational Learning: is to evaluate the extent of organizational and individual support and participating in organizational learning, e.g. whether an organization establishes and applies an Organizational Learning System. (7) Information Exchange Reporting System: is to evaluate the status of developing and operating an information reporting system in an organization, e.g. whether they have and operate a voluntary reporting system. Information Submission: is to evaluate an individual s desire and the actual behaviour of reporting safety information. Information Feedback: is to evaluate the status of information feedback from decision-making and management levels of the organization. Experience Sharing and Communication: is to evaluate the status of exchanging and sharing safety experience and information among individuals. (8) Rewards and Punishment Justice and Equality: is to evaluate the just and equal extent of establishing and executing reward and punishment (where deliberate violations have taken place) mechanisms in an organization. 3.3 Determination of index weight The next step of evaluation is to assign weight value to each index accord to their importance. The Delphi method was used to calculate the weight. The main steps are as following and the calculation method of rating score is shown in Table 1. (1) To design expert questionnaire according to the proposed index system, and 10 experts who had the professional background of safety were invited to complete the questionnaire and give scores to each index basing on their own knowledge individually. The scoring scale was set as 5 levels from unimportant to utmost important. (2) To make statistics analysis on scores. We set expert s score on index as,,, means the series number of experts, means the series number of indices. The calculation method is as follow table. (3) To deliver all data to each of experts anonymously. Experts will amend their judgments and scores again after comparing their scores with other ones. (4) To repeat the third and the forth step until the dispersion is less than or equal to the validity index. (5) To calculate the weight of indices in each layer of system. Table 1. The calculation method of expert rating score Index Description Expert: Mean: Weight: Dispersion: Finally their ratings were statistically analyzed and the final result of index weightings is presented in Table 2.
336 Lei Wang and Ruishan Sun / Procedia Engineering 43 ( 2012 ) 331 337 Table 2. Weight of safety culture evaluation index u i First-level indices Weight u ij Second-level indices u 1 Priority 0.157 u 2 3.4 Reliability analysis of index 0.155 u 3 Flexible 0.139 u 4 Learning 0.159 u 5 Teamwork 0.127 u 6 Reporting 0.136 u 7 Just 0.127 Weigh t u 11 Guideline and Policy 0.244 u 12 Objective and Responsibility 0.267 u 13 Organization Setting and Function 0.233 u 14 Resource Allocation 0.256 u 21 Completeness and Applicability 0.479 u 22 Executing and Fulfilling 0.521 u 31 Risk Management 0.379 u 32 Staff Involvement 0.273 u 33 Risk-resisting Capability 0.348 u 41 Quantity and Variety 0.205 u 42 Quality and Effect 0.261 u 43 Operational Skill 0.247 u 44 Organizational Learning 0.287 u 51 Teamwork Skill 0.585 u 52 Experience Sharing and Communication 0.415 u 61 Reporting System 0.369 u 62 Information Submission 0.309 u 63 Information Feedback 0.322 u 71 Supervision 0.506 u 72 Justice and Equality 0.494 A complete safety culture questionnaire containing 32 questions was designed basing on the developed evaluation index system. 123 copies of the questionnaires were distributed to a local aviation operator in Tianjin, China and the data was used to examine the reliability of the evaluation index (the consistency of a set of tests or questionnaire items). The obtained alpha coefficient, using SPSS, of the 20 items is 0.856 which indicated that the scale has adequate reliability. The overall reliability (Cronbach's alpha) of the survey varies from at the lowest 0.833 to at the highest 0.856 if one of the items is deleted. None of the values is greater than the current alpha of the whole scale (0.856), so it is not necessary to delete any of the items to improve the reliability score of this scale. 4. Conclusions The process of developing a new evaluation index system based on Integrated Model (ISCM) was examined. This model analyzed safety culture from two levels of intrinsic and extrinsic level. The potential relationship between the safety sub-culture and all of the safety culture dimensions was illustrated in this model. Therefore it provided the possibility of evaluation safety culture from these two perspectives. After that two forms of safety culture evaluation index system were constructed and every index was described. Then the weights of index were determined by using Delphi method and the reliability of the index system was also examined based on survey data. The results indicated that evaluation index system was effective and applicable, and it is expected to be as the basis of developing final safety culture quantitative measurement tool. In the continued work, a fuzzy comprehensive evaluation solution on safety culture was proposed out based on the output of this paper. The final safety culture measurement tool was also worked out and has been implemented in several aviation organizations in China and Europe. Acknowledgements We appreciate the support of this work from European Commission FP6 project of HILAS (Human Integration into the Lifecycle of Aviation Systems, AIP4-CT-2005-516181) and National Basic Research Program of China (No.2010CB734105). We also appreciate the great help of Dr. Marie Ward at Trinity College Dublin for revising the manuscript.
Lei Wang and Ruishan Sun / Procedia Engineering 43 ( 2012 ) 331 337 337 References [1] Cooper M. D., 2000. Toward a Model of, Science 36, pp. 111-136. [2] Reports Series No. 11: Developing in Nuclear Activities,Practical Suggestions to Assist Progress, International Nuclear Advisory Group, 1998. [3] Stanton, N. A., P. Salmon, et al., 2009. Predicting pilot error: Testing a new methodology and a multi-methods and analysts approach, Applied Ergonomics 40(3), pp. 464-471. [4] Reason, J., 1990. Human error, Cambridge, Cambridge University Press. [5] Helmreich, R. L., Merritt A.C., 1998. Organizational culture, in at work in aviation and medicine R. L. Helmreich & A. C. Merritt Editor. Brookfield, VT: Ashgate, pp. 107-174. [6] Wiegmann, D. A., Shappell, S. A., 2003. A human error approach to aviation accident analysis: The human factors analysis and classification system. Burlington, VT: Ashgate, U.K. [7] Von Thaden, T. L., Hoppes, M., Johnson, N., Li Y., Schriver, A., 2006. The perception of just culture across disciplines in healthcare, Health Care Technical Group- The Proceedings of the 50th Annual Human Factors and Ergonomics Society Conference, San Francisco, CA, p. 352. [8] Pasman Hans J., 2008. Developments in Science and Technology, Progress in Science and Technology (Vol.7) Part A- Proceedings of the 2008 International Symposium on Science and Technology, Beijing, China, pp. 3-12. [9] International Civil Aviation Organization: Management Manual (SMM), Document No.9859, Montreal, 2006. [10] Cox, S., Cheyne, A., 2000. Assessing safety culture in offshore environments, Science 34(1-3), pp.111-129. [11] Smith, G. S., Huang, Y.-H., Ho, M., Chen, P. Y., 2006. The relationship between safety climate and injury rates across industries: The need to adjust for injury hazards, Accident Analysis & Prevention 38(3), pp. 556-562. [12] Wiegmann, D., Zhang, H., von Thaden, T., Gibbons, A. and Sharma, G., 2004. culture: An integrative review, International Journal of Aviation Psychology, 14 (2), pp. 117-134. [13] McDonald, R. P., 1999. Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum. [14] Reason, J., 1997. Managing the Risks of Organizational Accidents, Hants: Ashgate, U.K. [15] Xu Deshu, 2007. Reform and opening, safety development and safety culture, Proceeding of China Occupational and Health Association 2007 Annual Meeting, Hangzhou, China, pp. 6-9. [16] Wang Lei, Sun Ruishan, Liu Hanhui, Zou Mei, 2009. Integrated Model and its Application, Journal of Civil Aviation University of China 27(4), pp. 22-25.