Develop Performance Measures

Transcription

1 Develop Performance Measures A performance measure is used to measure output or outcome, such as assessing the impact of corrective actions on safety. They enable stakeholders to monitor changes in system condition and performance against the Corrective Actions (CAs), and drive improvement. Benefits Increased organizational focus on safety priorities A better understanding of the impacts of alternative courses of action aimed at improving safety Feedback to promote ongoing improvement as they relate to supporting safety strategies Greater accountability and improved communication about safety to stakeholders Come Up with a Useful Performance Measure that You Can Actually Use to Collect Data Once we have identified the problem, the next challenge is to come up with one or more performance measures to determine whether our implemented corrective actions are effective. Now you might be thinking that the next step should be to come up with our corrective actions and then the performance measures will follow from there. The problem is that in coming up with performance measures after the identifying corrective actions, we often focus on process rather than impact. So if we come up with a training module as our corrective action we may use the number of people who get trained or how many hours of training they receive -- metrics we can easily measure. However, this is not a measure of impact or outcome. By coming up with a performance measure before we identify our corrective actions we are more likely to focus on outcomes. So we measure the number of safety behaviors our training class produces. If we can t come up with performance measures this tells us we have not dug deeply enough to identify the source of the problem. For carriers already using safety performance measures, these steps can help identify new measures and/or enhance existing ones. If performance measures are not currently used, these steps can help identify, select, and incorporate them into the transportation planning process. The person responsible for the measure should be in a position to take action, if the measure suggests that that is what is required.

2 Develop Performance Measures: Establish where performance gaps existed that resulted in the close call. Based on your assessment of performance gaps contributing to the root causes you selected, determine actions that should be implemented to resolve the gap. How Do We Define a Performance Measure? Set performance goal. How should performance change? Define target performance List the observable behavior(s) to measure Measures should be relevant, be controllable and have a defined (significant) impact on the required outcome that is measurable within a relatively short time scale. Performance Measures can vary by carrier Because each carrier has different ways of managing its business, there is variation in how each railroad manages its risk. Many companies separate management subsystems, such as business processes, occupational safety, system safety, etc. The fragmentation of risk responsibilities can contribute to inefficiencies and can badly influences measures, such as safety and cost. Many carriers lack a mechanism to integrate different kinds of risks that it manages. They may also lack a system for tracking and monitoring safety-related corrective actions. Steps 1. Develop evaluation criteria. Define the performance that you will measure to determine if your CA worked (e.g., Ability to detect a switch) 2. Define performance measures to evaluate impact. Identify the behavior do you want to change. How should performance change? What performance do you expect to see if your CA is successful (Increase the distance at which employees detect a switch or the percentage of employees that will detect the switch)? State the goal as an outcome rather than an action. A goal should be results oriented, specific, address a single result, it can be improved (changed), aligned with other safety improvement goals. To be concrete and specific you should be able to see, hear, feel or smell something? List the observable behavior(s) you can measure. Use example of detecting as switch. How far away can the train crew detect the switch or switch target Training: Observe employee s behavior following training. Count # of safe behaviors Rule compliance: Observe percentage of people complying with a rule change Change in paperwork. Determine how quickly employees can find pieces of information. Measure whether they find the correct information. Estimate the strength of your measure. The stronger the measure, the more closely tied it will be to your underlying problems. Is your measure feasible? How easy is it to collect this data? Is the data already available? If no, how much effort is required to collect this data? 3. Measure the problem: Collect baseline data. Collect other carrier data as needed. To get performance measures look at carrier data, local site data and FRA data. Does the carrier collect data on the proposed measure(s)? Will new sources of data be needed? Present carrier s existing operational safety data by Safety Department data expert at their organization. The FRA PRT member (assuming that they have been trained in accident investigation methods) can teach & facilitate the investigation of on-property investigations by 2

3 other PRT members, or others within the railroad. Carrier s Safety Department data expert facilitates the investigation of on-property investigations by other PRT members, or others within the railroad. Look at examples of measures to get ideas. Avoid the natural inclination to look only to readily available measures. Collect right data and organize data Determine the number of measures. Multiple measures are often impractical. If multiple measures are used, conflicts between measures may occur and must be resolved. 4. Identify what performance level represents success: To determine if the problem has been solved, define performances to measure the impact of the corrective action. This step must occur before developing the corrective action. By identifying the performance measures before developing the corrective action, we reduce the temptation to measure compliance with the plan instead of the plan s impact on the problem. If you cannot define performance measures to evaluate the corrective action, then the problem identification process needs more work to make the problem manageable. After the performance measure has been identified, identify what performance level represents success. Describe how the performance will change with implementation of the corrective actions. Define the target performance that you will use to indicate whether performance has met your goal (e.g., Percent of employees that can detect a switch from xxx feet from the switch, Distance from switch). Define how we will recognize success if the Corrective Action is effective both short-term and long-term? What does the safety outcome look like when a Corrective Action has fixed a problem? Include human factors aspects. Using benchmarks to set targets and measure performance Previous performance (i.e., use own data) and show baseline Performance of similar carriers Established or recommended standards (Federal, state, third-party recommendations) 5. Identify data needed for Safety Performance Measures. Types of Performance Measures Quantitative measures are tangible, numerical representations of goal achievement (e.g., number of units, outcomes). They are easy to understand and get results. Qualitative measures are intangible behaviors and performance. It is a non-numerical representation that is narrative or pictorial in nature. Strategies require that organizational roles and responsibilities, needed resource allocations, and data collection methodologies are incorporated into the standard procedures of the carrier. Data sources must be credible. If credible data is not available for a measure, then the measure is not feasible. Data sources must be cost effective. Minimize costs by using existing data, using random samples rather than collecting data on every instance A careful examination of the collection strategy should be made to assure the data meet the quality standards and schedule needs to support the performance measures. In some case, collecting the desired information will become part of the CA process. Review measure types for-quality, quantity, accuracy, timeliness and cost. 6. The following characteristics are critical data collection factors for consideration as part of an overall data management strategy: Accessibility data need to be accessible for safety analyses. 3

4 Accuracy the data need to be accurate, whether as originally collected or in their final electronic format. Completeness crash data records should include all crash characteristics so the analyses incorporate all factors and improve understanding of the contributing factors. Uniformity/consistency uniformity and consistency in the data base is critical to assure consistency in reporting. Timeliness measures the delivery, of expected results according to schedule. The data system custodian should produce data in a timely manner to inform the performance measurement process. Data also needs to be updated regularly. Integration databases should be developed to allow the compatible transfer of data from one database to another. 7. Determine if data supporting the performance measure exist. Consider factors when developing indicators, including: Format of the data Frequency Method of data collection Time Context and analysis Users and uses of the data Costliness of collecting the data Example: Selection of Performance Measures and Corrective Actions Exceeding speed limits through temporary speed restrictions Summary: The engineer was operating a 1 mile long freight train through track warrant territory with multiple temporary speed restrictions (TSRs). The engineer exceeded the speed limit by 10 mph while moving through a temporary speed restriction of 25 mph. The engineer detected the speed violation when he saw the track sign marking the end of the temporary speed restriction. The engineer was traveling from west to east and missed the temporary speed restriction identified in the train bulletins for that trip. Causal factors 1 The train bulletins were organized so the engineer could not easily find the TSR as he made the trip. The TSRs were always organized from East to West, so that the engineer had to read the paperwork starting from the last page and move up the page toward the first page. The milepost locations and the TSR were embedded in many closely spaced lines of text that made it difficult to find. 2 Signs or flags marking the beginning of the TSR were missing. The signs or flags were also difficult to see at night as they were located outside of the field of view of the headlight and were often faded. 3 The engineering department spaced multiple TSRs closely together (less than a mile apart) making it difficult to increase speed for more than a short time before having to reduce speed. For long trains (those one mile long or greater), this condition created a trap where engineers speeding up in between TSRs could not slowdown in time to stay within the maximum speed in the next zone with a TSR. Performance Measures and Criteria Performance Measure Method Criteria 1 Time to locate TSR and Speed information Speed ID accuracy 2 Deviation of sign from track right of way Measure time for conductor to find and report speed for TSRs during train operation Measure accuracy of conductor to find and report speed for TSRs during train operation Measure extent to which trains exceed TSR Measure distance of sign/flag from track 80% of items found within 2 seconds. 95% of items reported correctly. Sign/Flag position 10 ft. or less from track 4

5 Exceeding speed limits through temporary speed restrictions Sign luminance 3 Distance between TSRs Proposed Corrective Actions Measure sign luminance for distance at which sign must be detected for engineer to respond to TSR Measure extent to which trains exceed TSR Measure distance between TSRs and compare to train length. Minimum distance should be at least Measure extent to which trains exceed TSR. Sign luminance at max speed of 55 mph is 250 cd/m2 100% TSRs separated by distance equal to twice max allowed train length (1.1. miles) 1 Write train bulletins so TSRs are organized by the order in which the train will pass them. If trains travel east to west, organize bulletins so that TSRs go from east to west by milepost location. Information should be organized so that they start at the top of the first page and go down the page. Organize information in a tabular format and highlight safety critical information, such as milepost location and speed information. 2 Set signs or flags at the beginning and end of all TSRs to indicate when they begin and end. Locate all signs/ flags no more than 10 ft. from the track right of way. On right of way with multiple tracks, place the signs/ flags in the same positions relative to each track, so the engineer can associate the correct sign /flag with each track. Each sign should use reflective material with a minimum luminance of xxx cd/m2 3 Position TSRs so that the minimum distance between any two TSRs is at least 2 train lengths. Incorporate Safety Performance Measures into Planning 1. Set baselines and measure performance regularly. 2. After the candidate performance measures have been considered, those that are selected need to be incorporated into the planning process. Monitor and Evaluate Effectiveness: Assess the completeness of your root cause work against the organization performance factors (PFs) listed. If you have missed any PF, circle back to reassess, making sure you either are satisfied with the factor s activities or filling in gaps. 5