Using the SMART Method to Assess SMS in Aviation

Transcription

1 CASS COUNTING THE ACCIDENTS YOU DON T HAVE Using the SMART Method to Assess SMS in Aviation Ronald J. Heslegrave,, Ph.D. University Health Network & University of Toronto May 1, 2007 Gatineau QC

2 Successful Implementation of a Safety Management System or Policies Need to: Develop Develop a Vision for the Organization without a culture it will fail Develop Develop Policies and Procedures designed to Support a Safety Culture and the Vision Implement Policies and Procedures through Training, Organizational Strategies, Operational Strategies, Enabling Tools Develop Develop Assessment and Evaluation Mechanisms Evaluate Evaluate what you have put in place

3 The Question sometimes is How do you do that?

4 One Answer Adherence to Recognized Standards or Adopted Regulations where there is consensus that such Standards or Regulations reflect an appropriate Vision and Policy for example, ISO standards

5 Another Answer Evaluation of objective measures where most agree that the measures reflect the Vision and Policy for example, reduced accidents, incidents or near misses

6 Near Misses Or Worse

7 Another Answer Accept expert testimony and evaluation that the Vision and Policy have been achieved in good measure for example, Expert External Audit

8 Auditors and Regulators are sometimes viewed differently than intended It depends on your perception?

9 Auditor and Henchmen may be viewed Negatively

10 The Audit Process can be viewed as Confrontational

11 We should strive for a Reasonable Dialogue with Enlightened Views between Auditors and Organizations

12 Is there a simple, effective and multidimensional approach to evaluating SMS in your organization that is valueadded?

13 Perhaps? Today I will introduce the SMART approach to safety management and evaluation? While this approach has been investigated in the civil Aviation sector, it should be transferable to other organizations and systems

14 Safety Management Audit & Review Tool- SMART method Evaluates the Understanding of Fundamental System Elements of a Safety Management System Evaluates Supporting Systems: Training and Measurement, Assessment and Evaluation Tools SMART method evaluates key elements of safety systems or policies Assesses whether the Safety Management System has Penetrated the Organization at all levels

15 Safety Systems and Policies are Multidimensional - Evaluations Need to be as well Assess the reasonableness of what is proposed Assess how well it is implemented Assess whether it is effective Develop a system for continuous quality improvement to monitor and promote change Feedback the evaluation to the Organization to provide continuous quality improvement

16 BASIC EVALUATION APPROACH OF THE SMART METHOD Fundamentally at least 3 Criteria or Dimensions: Understanding of the Policy and Procedures Quality Quality of Implementation of the System Penetrance of Safety Policy and Procedures throughout the organization

17 SMART METHOD 1) Identify the goals of the system 2) Develop a set of tools designed to target the primary requirements of that system 3) Collect objective and subjective data from relevant stakeholders across the hierarchy of the organization 4) Blindly evaluate and analyze these data in a multidimensional context 5) Provide feedback for ongoing quality improvement for the organization

18 Key to the approach is to Quantitatively Assess and Evaluate the Data collected on a Multidimensional Basis

19 Can this SMART method be implemented Practically, Easily and Successfully?

20 You be the Judge?

21 An Example of the Application of the SMART Method Recently the SMART method was validated in a major corporation in the Aviation Community with national operations to study the successful implementation of their Safety Management Program and Policies

22 SMART APPROACH Identified Identified Key Policies and Objectives of the Safety Management Program Developed an Assessment Team Developed a Survey to Assess Safety Management Implemented a multidimensional approach to the evaluation of written and interview responses to the Survey Tool

23 Expert Team Analysis An Audit Team was established with varying expertise including Operational Systems, Safety Management Operations, Specific Content Experience, and Scientific Expertise Prior Prior to assessing the data collected, a decision was made to blind the evaluators to each others scoring of responses with regard to all aspects of the evaluation

24 Develop a Survey Tool Questions were grouped into Common Safety Dimensions Questions were developed along 5 Dimensions representing the Continuum of Safety Management Systems Fundamental Program Elements Implementation of Fundamental Program Elements Training Assessment and Evaluation Strategies Measurement Systems

25 Collect Data from Organization Submit the Survey Tool to the Organization for a Formal Written Response Based on the Survey, Interview Key Stakeholders across the Hierarchy of the Organization Executive Senior Management Line Managers Safety Professionals Line Personnel

26 Develop a Multidimensional Evaluation Strategy Quality Quality of the Response - How well do the answers address the questions asked? Policy Policy Understanding - How well do the answers demonstrate an understanding of the Safety Management Policies? Penetrance - How well does the answer demonstrate the implementation of the Safety Management Policies throughout the organization?

27 Evaluation Process Each Each Audit Team Member Independently and Blindly assessed each written and interview response All ratings were made on a 5-point 5 Likert Scale: 5= Good and Full Response 3= Reasonable Response 1= Inadequate or Low Response

28 Organizational Feedback Results are presented in the following fashion: Overall ratings of Quality, Policy, and Penetrance Dimensions Ratings on a question by question basis to identify areas of strength and opportunity for improvement in the Organization Summarized data by common dimensions based on the questions

29 Findings

30 Overall Ratings of Audit Team High 5 4 Reasonable 3 2 Low 1 Written Questions Interview Questions Average QUALITY POLICY PENETRANCE

31 Overall Ratings of Audit Team for Written Responses High Reasonable Opportunities for Improvement 2 Low 1 QUALITY POLICY PENETRANCE

32 Ratings of Audit Team for Written Responses who were blinded in their assessments High 5 4 Reasonable 3 2 Low 1 Auditor 1 Auditor 2 Auditor 3 QUALITY POLICY PENETRANCE

33 It is often said that: The Devil is in the Details In this case The Added Value is in the Details

34 Individual Question Results High 5 4 Reasonable 3 2 Low 1 0 Q1 Q2 Q3 Q5 Q6 Q7 Q8 Q9 Q10 QUALITY POLICY PENETRANCE

35 Individual Question Results High 5 4 Reasonable 3 2 Low 1 0 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 QUALITY POLICY PENETRANCE

36 Individual Question Results High 5 4 Reasonable 3 2 Low 1 0 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27 Q28 QUALITY POLICY PENETRANCE

37 Individual Question Results High 5 4 Reasonable 3 2 Low 1 0 Q29 Q30 Q31 Q32 Q33 Q34 Q35 Q36 Q37 QUALITY POLICY PENETRANCE

38 Individual Question Results High 5 4 Reasonable 3 2 Low 1 0 Q38 Q39 Q40 Q41 Q42 Q43 Q44 Q45 Q46 QUALITY POLICY PENETRANCE

39 The SMART Process: Analysis of Individual Questions The The results of individual questions reveal areas of excellence as well as specific opportunities for improvement: Questions 4 and 5 show company-wide excellence

40 SMART - Individual Questions High 5 4 Reasonable 3 2 Low 1 0 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 QUALITY POLICY PENETRANCE

41 The SMART Process: Analysis of Individual Questions Questions 8 and 9 show strong enabling tools and good penetrance despite less than adequate policy description - perhaps due to self-evident evident data

42 SMART- Individual Questions High 5 4 Reasonable 3 2 Low 1 0 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 QUALITY POLICY PENETRANCE

43 The SMART Process: Analysis of Individual Questions Question 2 shows good policy development but poor penetrance throughout the organization Question 1 and 3 indicate opportunities for improvement

44 SMART - Individual Questions High 5 4 Reasonable 3 2 Low 1 0 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 QUALITY POLICY PENETRANCE

45 Rather than looking at individual responses to written and interview data, another approach was to aggregate the information into Safety Processes. Data was aggregated following processes: Fundamental Requirements Training Processes Evaluation Processes Measurement Processes Implementation Processes

46 Ratings of Responses by Processes Fundamentals Training Evaluation Measurement Implementation Low Reasonable High Penetrance Policy Quality

47 In general these data show a strong emphasis on training and implementation

48 Ratings of Responses by Processes Fundamentals Training Evaluation Measurement Low Reasonable High * Implementation * Penetrance Policy Quality

49 These data show that penetrance into the organization falls behind policy understanding and an ability to articulate policy perspectives

50 Ratings of Responses by Processes Fundamentals Training Low Reasonable * High Evaluation Measurement Implementation * * * Penetrance Policy Quality

51 Evaluation strategies and measurement tools could use more emphasis

52 Ratings of Responses by Processes Fundamentals Training Evaluation Low Reasonable Below acceptable High Measurement Implementation Mostly Below acceptable Penetrance Policy Quality

53 Interview Findings

54 Generally Safety Personnel Understood Policy and Could Articulate the Policy but Generally Agreed that the Penetrance in the Organization could be improved

55 Ratings of Interviews for Managers and Safety Professionals High Reasonable 3 Low 2.5 Senior Managers Safety Professionals Line Managers QUALITY POLICY PENETRANCE

56 Comparison of Interviews and Written Responses High 5 4 Reasonable 3 2 Low 1 Senior Managers Safety Line Managers Written QUALITY POLICY PENETRANCE

57 Ratings of Interview Questions High 5 4 Reasonable 3 2 Low 1 0 Q25 Q26 Q27 Q28 Q29 Q31 Q32 Q33 QUALITY POLICY PENETRANCE

58 Ratings of Interviews by Processes Low SMS Daily Duties Reasonable High Measurement of Safety Contractor Training Economic Impact on Safety * Penetrance Policy Quality

59 Are there examples of questions you can ask in your organization? You bet!!

60 Example 1 Non-Punitive Reporting Systems Does the company have a non-punitive reporting policy? If so, can you describe it and tell me if it is followed in day-to-day operations?

61 Non-Punitive Reporting Systems: Answer The company has a well-documented and comprehensive non-punitive reporting policy that can be found in the flight operations manual and is posted on all safety bulletin boards. The policy includes easy to use web-based based and paper reporting forms, and has an option for anonymity if the reporter requests it. Unfortunately, most people won t fill out reports because they don t believe they will be protected from prosecution by management if they report errors.

62 Non-Punitive Reporting Systems: Scoring Quality of Response = 3 The answer is of reasonable quality and demonstrates a fundamental understanding of the company s non-punitive reporting policy Policy Assessment = 4 The answer reflects a good understanding of the non-punitive reporting system, including details such as where it can be found and how to use it Organizational Penetration = 1 Even though the answer is of reasonable quality and demonstrates a good understanding of the policy, it shows poor adherence and the policy has not penetrated into day-to to- day operations

63 Example 2 Reactive and Proactive Safety Reporting What processes or steps do you follow when you have an incident and observe a hazard? How do you think it is working?

64 Reactive and Proactive Reporting Processes: Line Staff Answer When there is a safety incident, there are specific procedures that we follow in the operations manual and forms that we fill out. [Prompt, how do you think this is working] It works pretty well I guess but I don t get much feedback. I used to fill out the forms for everything but now the incident needs to be serious before I do the paperwork. When I observe potential hazards I usually let my supervisor know.

65 Reactive and Proactive Reporting Processes: Line Staff Good response and understanding of Policy and Procedures Good adherence to procedures but becoming diligence seems to be reducing over time indicating poorer Penetrance Much better Policy and Procedures for Reactive Reporting of Incidents than for Proactive Assessment of Hazards

66 Reactive and Proactive Reporting Processes: Executive Answer When there is a safety incident, there are specific procedures that t need to be followed in the operations manual and forms that we fill out. All employees are trained on these procedures and forms are completed and analyzed so that corrective action can be taken. These reporting procedures are flexible enough to cover both reactive and proactive reporting and are used in both cases. [Prompt, how do you think this is working] It works very well. We have binders of analyses and act on these analyses to improve safety. It must be working well because the reports pick up data on serious incidents. Safety is improving as demonstrated by fewer reports every year.

67 Reactive and Proactive Reporting Processes: Executive Good response and understanding of Policy and Procedures No demonstration of the need for feedback to ensure maintenance of implementation Declining Penetrance given that line staff are restricting their reporting and the Executives interpret this as improved safety.

68 Summary The SMART method produced objective assessments of the Safety Management System The approach appears to provide reliable ratings and consistent results The findings appear to identify areas of excellence and opportunities for improvement

69 Summary Clearly Clearly there are differences in the Quality of Responses, Understanding of Policy, and Penetration of Safety into the Organization that need to be addressed The The approach clearly identifies strengths and weaknesses in the Safety Management System to help focus Continuous Quality Improvement with regard to Safety Management Systems

70 Summary Importantly, the feedback from this analysis provides the organization with a baseline against which to measure change (improvement) of future Quality Improvement Strategies and changes in subsequent Policy Development

71 Summary Equally important is that this type of feedback helps target resources to strategic areas Using this strategy, decisions can be made empirically on the most important areas to invest resources while protecting resources for areas where accomplishments have occurred

72 Summary This This approach will allow strategic decisions This This approach will allow the strategic allocation of resources to where they might be of most benefit This This approach needs to be informed by future applications leading to a manualized approach that can be widely available with appropriate consultation

73 Multidimensional Conceptual Model of Successful SMS implementation High Penetrance Low Minimum Success High Low Policy Understanding Low High Quality of Response

74 Conclusion ScientificScientific Defensible ScaleableScaleable Performance Based Continuous Quality Improvement Tool

75 Conclusion Allows Allows self evaluation Comprehensive Simple Simple to use Theory Theory driven - Practically Oriented Transparency

76 President & Chief Executive Officer on the SMART Method the overall process was helpful in identifying the method of conducting a safety audit [our organization] can learn significantly from this process.