Session No. 574 Balancing Your Safety Metrics Why Leading Metrics May Be Misleading Paul Esposito, CSP, CIH President, STAR Consultants, Inc. Introduction Using data collected from leading companies worldwide, more and more are moving away from incidence rates as their sole means of measuring safety performance. In this session, attendees will Identify leading metrics from a Balanced Scorecard perspective Learn to avoid the common pitfalls and Learn implementation strategies for leading metrics Historically, incidence and severity rates have been the key metrics for measuring safety program success, and in part, rightly so. However, as we have learned from Kaplan and Norton of Harvard, the creators of the balanced scorecard, no one number should be used to measure the strategic implementation and success of anything. In addition, many leading companies have been branching out from the historical lagging metrics, and initially found mixed success with the leading metrics they chose. Successful manufacturing companies have been surveyed, and the results of that survey will be presented in this session. In addition, their lessons learned, how some leading metrics did not meet expectations, will be discussed. Foremost, the concept of the balanced scorecard, four quadrants being measured simultaneously to provide a sort of check and balance to strategic initiatives, was misunderstood. Safety metrics were seldom developed for all four quadrants. Likewise, leading data collection systems were almost universally lacking, making both the collection and analysis part of the strategic initiative. Hear how leading companies are using multiple leading metrics from each of the four balanced scorecard quadrants to measure safety performance. Insights into their lessons learned, and future metrics will be presented as well as implementation strategies for weaning management off of just incidence rates.
So, what were some of the common metrics? Within the Process quadrant, almost all leading companies had some sort of management systems audit with scoring, among others. A more recent metric is being derived from risk assessments. From the Customer quadrant, most did some sort of perception survey, but some were maturing into measuring employee engagement. From a People quadrant, training attendance was the typical starting place, but all soon realized that attendance did not necessarily translate into learning or improved behaviors. Finally, from the financial quadrant, metrics other than worker s compensation needed to be measured. Many have turned to budget implementation, manpower and other leading metrics. This presentation will walk through each of the four balanced scorecard quadrants, present what metrics these leading companies started with, added and are currently using. Common pitfalls and lessons learned will be explained. Leading Metrics with a Balanced Scorecard For over 50 years now, Deming and others have extolled the value of upstream metrics, process metrics as a means of driving bottom line results. If you measure the process, results will come is a famous Deming quote. Therefore, successful companies recognize that a balance of leading and lagging metrics, sometimes knows as a Balanced Scorecard 1, where Financial, Process, Growth and Learning and Customer metrics are combined to better measure both the progress of a company or program, as well as the status. In addition, Dan Petersen, considered by many to be the guru of safety management, published 2 a list of 6 essential and suggested metrics, which are included in the list below. Key Financial Metrics: Worker s Compensation: While not consistent from state to state, country to country, it is one measure to keep track. (Petersen) Manpower: what is the amount of time allocated to safety form among the organization, and does it get used. Budget: is there a budget to implement corrective actions, and does it remain throughout the lifecycle of the year. Key Process Metrics: Safety Management Systems (SMS) SMS Site 1 Site 2 Site 3 Site 4 Assessment Scores. (Peterson) Score 45 65 75 86 Leading companies like BorgWarner, GE, % Closure 10 30 45 80 Honeywell, Standard Aero, Eaton, etc. use Score based on 100%. >80 = green. <50 = red % Closure: >50% yellow, 100% = green the Key Element Audit (Management Systems) process as a means of measuring process. We see this with ISO 14000, OHSAS 1 Kaplan and Norton, Harvard, 1995 2 Occupational Hazards Magazine, March 2001
18000, ISO 9000, etc. If the audit process is sound and substantial in its verification of process (not just documentation) it is valuable not only as a means of reporting accountability and process improvements, but as a tool to help operational organizations know where to improve next. Management systems improvement initiatives. (Petersen) Assessments typically generate improvement initiatives. The timely closure of these initiatives are also tracked as part of a global scorecard. Risk Reductions. Moving the metrics from an accident based to # of Moderate Risks a risk based approach is also more in line with Site 3 what the current trend is towards World Site 2 Yr Class. Risk reductions are measured by a number of different means. Depending on the Site 1 prev Yr data available, some just measure the number 0 5 10 of residual risks, and its trend, based on initiation of safety improvements. Others measure the upstream data, i.e., the number of corrective actions implemented that had elimination, substitution or engineering controls. Another prospective metric is the reduction in people required to wear PPE. Key Learning and Growth Metrics Training completion and retention. Training completion by itself is a lagging metric. It does not factor in quality of the instructor or learning that occurred. Often, as part of an inspection or observation program, knowledge based quizzes are incorporated to generate a retention metric. Conformance Rate to significant controls (based on inspection and observation data). A conformance rate is the 5 of safe vs at-risk. Either conditions, behaviors or both, more specifically the use of controls, be they engineering or work procedure. Significant risks are identified as are their controls. The conformance of these specific controls are calculated as a Conformance Rate as a result of site inspection and observation programs. Typically targets of 95 100 % are expected. (Petersen - Behaviors) (Compliance): this is an optional leading metric typically generated from some type of facility audit, typically performed by the site itself with 3 rd party audit verification. Most often, compliance will be verified for significant high risk exposures. Corrective Action Closures: regardless of the source of corrective actions. Corrective actions could come from inspections, observations, investigations, industrial hygiene surveys, risk assessments, etc. Number of new engineering, substitution and elimination controls: regardless of whether these come from risk assessment, incident analysis, etc. Key Customer Metrics Culture: typically performed as part of a survey. (Petersen) culture is often defined as the way we treat each other (Daniels.) He says, the elements of culture, getting people to do what we want them to do, is that positive reinforcement is much more effective in getting
people to do what we want, rather than punishing what we don t want. Therefore culture usually measures four areas: 1) Management Leadership and Commitment, 2) Employee Engagement, 3) Communications and 4) Resources within the Organization. The culture of an organization dictates the rigor to which things get done. It is the volume of positive reinforcement and recognition vs a punishment and penalty mentality. TIR: Total Incidence Rate: This is a lagging metric (indicator, actually) of injury frequency. (Petersen) DART: Days away and restricted case rate: This is a lagging metric (indicator) of severity. These are the two significant metrics of OSHA s VPP (Voluntary Protection Program), along with the SMS audit and closure rate of SMS action plans. Employee involvement and engagement: are your workers participating in meaningful ways as part of your safety programs. While many of these metrics take time to develop (generate and verify data) an audit program is the typical starting point, as then there is a mechanism in place to begin to verify some of the other metrics. The audit also tends to be done by an independent 3 rd party, at least as part of the audit team, leading some objectivity the generation of metrics. The audit also generates some of the other metrics (Closure rate of management systems improvement initiatives). You may note that the Incidence Rate and Days Away metrics are labeled as indicators. This means that they are not a true measure of anything process-wise; certainly not a direct measure of your customer. However, they are indicators that are typically monitored to track against other metrics in the balanced scorecard application. Common Pitfalls Measuring for the sake of measuring is often a recipe for metrics to lose value; quickly. In addition, focusing on the negative, like incidence rates, does not measure what you want people to do about prevention. John Teimeyer, corporate safety and security director for an airplane maintenance company, says we measure things that we want, rather than what we don t want. This helps management focus on those things that matter most. If delivered correctly, he says it empowers and enables leaders to evaluate their respective operations and implement measures that work for them. Corporate / Site / Organizational Level Another common pitfall is having the same metrics are each of the various levels within the organization. While they do benefit from cascading up and down, the level of detail becomes more significant at the lower (site and organizational) levels. This does not mean, however, that organizations within sites are accountable to metrics when then have little or no influence. For example, the overall SMS scores are influences by many parties. Therefore, at the organizational level, they are held accountable only to the SMS action Plans, not the overall scores. The scores are used at the site and corporate levels. At the site or organizational level, typically and additional level of detail is need to better monitor, review and react to the various safety programs and initiatives. Specifically, the amount of change and closure are the key data points, along with some measure of quality. In terms of
risk assessment, this is one of the programs which yields a full breadth of metrics. Initially the number of assessments can be a target, but to be worth the investment of time, changes needs to be identified. Otherwise it was just a paper exercise. These changes need to be closed and stay closed, so some organizations add a 30 day ticker to the closure as an additional metric. Finally effectiveness is based on the type of control that gets implemented. More effective controls include substitution, elimination or engineering controls, rather than just training or personal protective equipment. Whether the target is a number or a percentage, defined or just compared to the previous reporting period, comparison against a target is essential. Leading Metrics Can Be Misleading As mentioned earlier, Kaplan and Norton are adamant that no one number should be used to measure any significant program or process. Thus, for each program that is important to measure, we measure 1) numerous metrics and 2) things that we want people to do. Often, just the number of events is measured. While a leading metric, there is no level of quality or accomplishment. Therefore, multiple metrics for each program is a key concept. In this scorecard, there are targets for each metric. Achieve the target to get a Green. Get to within 10% or one number, yellow. Red is everything else below. Topics Current Program Risk Assessment Changes to Controls # of Events # of Changes Identified/Needed Closure Rate (From Date Closed) Effectiveness (Control Type or conformance) Actual Target Actual Target Actual Target Actual Target 5 20 4 5 60% 80% 23% 20% Monthly Safety Review - Incidents 8 10 10 5 75% 80% 10% 30% Monthly Inspections/Discrepancies high risk control conformance Monthly Observations (Optional) conformance or % safe operations Communication by Supervisors (ESH Tool Box) 20 10 5 20 96% 85% 93% 95% 50 10 20 10 96% 85% 93% 95% 10 20 9 10 5% 30% ESH Action Plan Status 5 20 85% 80% EHS Rewards and Recognition 50 30 Building up to this level of scorecard may take time. You may start out with only one metric in each Topic. To avoid common pitfalls, spread the metrics around by topic. For example, if you use the number of events for communications by Supervisors, you might want the number of new engineering controls to be your starting point for Risk Assessment. In reality, the starting points may actually be the data you have, rather than what you want, until such time as the data you want is available for collection. By no means is this the universe of potential data. Additional data can include: Employee involvement
Training completion and retention Suggestions Perception or Satisfaction Surveys, etc. There is a universe of additional programs, data and metrics that can be placed into a safety balanced scorecard. Implementation Strategies for Leading Metrics Successful organizations understand that there is no ONE metric for either the entire program or the individual component parts. A balance, or complimentary set of leading and lagging, or better, input and output metrics are used in addition to the standard lagging metrics like workers compensation and incidence rates to better drive continuous improvement. Likewise, this is not a race. This is a journey. Take the time to pilot metrics to gain acceptance, issue metrics when the data is sound, and you can tie success into recognition programs at the worker levels. As performance matures, metrics will need to change or expand. While we will also measure incidence rates and corrective action closure rates, the targets for number of new engineering controls may change (reduce) over time, as it becomes more and more challenging to engineer out processes. Another item is to get metrics into each of the four quadrants as soon as possible. Often times, management tells safety just to be in the customer quadrant. While this may be a good place to start, it does not match up to the philosophy of a balanced scorecard, where metrics are used within all four quadrants. If done correctly, with the right data, management will more often not only accept these new metrics but embrace them. Most of their other departments, like quality and production, have been using multiple metrics for years. Lastly, these metrics are then used as the measure of safety performance; line management is held accountable to these metrics as part of the performance appraisal process. Bibliography Daniels, A. PS Asks. Professional Safety Journal. September 2013. Esposito, P. Selling Safety to Management Using Metrics. Industrial Hygiene and Safety News. June 2002. Esposito, P. The Balanced Scorecard: A Powerful Tool for Risk Management. RM Insight. 2014. Kaplan and Norton. Using the Balanced Scorecard as a Strategic Management System. Harvard Business Review. 1995. Petersen, D. What is the best way to measure safety -- audits, incident rates or workers' comp cases? The answer may be "all of the above" and more. Occupational Hazards Magazine. 2001.