The Upsides and Downsides of Project Metrics JASON DUNN RESTON LUNCHEON PMI WASHINGTON DC CHAPTER NOVEMBER 15, 2017

Transcription

1 The Upsides and Downsides of Project Metrics JASON DUNN RESTON LUNCHEON PMI WASHINGTON DC CHAPTER NOVEMBER 15, 2017

2 Speaker Introduction: Jason Dunn Spent nearly 30 years in projects across a wide variety of industries, including 17 in Information Technology. Served in many different roles on projects from Project Manager, Change Manager, Business Analyst, and Tester. Managed small to medium-sized IT projects (>$30MM), serving as a program and portfolio manager as well. Worked with many Fortune 100 companies to assess and improve their project management maturity and processes through project risk assessments and system benchmarkings. As part of this work, I have stood up PMOs as well as assessed and improved existing PMO using a data-driven assessment framework. Led many different industry-sponsored studies in areas such as owner/vendor relationships, contracting, project recovery, labor productivity, knowledge management and design reuse. Recently developed a 2-day training program based on an industry standard maturity framework aimed at assessing, and then improving, companies portfolio development and management capabilities. Currently, I am the PM Community Program Director in Freddie Mac s EPMO, which is committed to improving PM maturity and capabilities through education, networking, facilitation, and knowledge management. PMP, CSM, CSPO, Certified Kanban Practitioner and ProSci Certified Change Manager.

3 Outline of Today s Discussion Discuss what metrics are and what they are typically used for Outline some dangers behind project metrics Provide tips on how to use or create metrics to achieve a goal Offer support for some metrics that you can use on your projects

4 Definition of a Metric A standard of measurement Merriam Webster A business-oriented definition: A standardized formula used to assess performance over time and monitor progress toward goals From Forbes Magazine Assess performance end results Monitor progress ongoing results A metric is typically represented by some form of statistic Metrics are typically meant to drive performance and, thus behaviors!

5 Why Use Metrics? There are three types of lies -- lies, damn lies, and statistics. Benjamin Disraeli Facts are stubborn things, but statistics are pliable. Mark Twain Without data, you are just another person with an opinion. W. Edwards Deming There is a need for some form of objective measurement for managing processes Otherwise, you cannot : 1) monitor the process, 2) figure out how to improve it, and 3) predict how it will perform While an issue could be happening After the project ends Before the issue starts

6 What metrics or information do you use to control or monitor your projects?

7 What would you do, as a project manager? Schedule Slip Our project is slipping its schedule and it appears that recovery to original deployment date is unlikely Complete a Variance Request to add four weeks to the schedule Investigate why the project is slipping and develop response plan Look for ways to reduce the time for other activities All of the above

8 Discuss the Control / Monitor Metrics Bad Good

9 What metrics or information do you use to improve your project processes?

10 What would you do, as a portfolio manager? Cost Deviation The average underrun of the projects in our portfolio was -14%, +/- 25% standard deviation Reduce budgets by 15%, across the board Look at what elements of the project were overestimated and lower their estimates Look at how the estimates were produced and change the process All of the above

11 Discuss the Improve Metrics Bad Good

12 What metrics or information do you use to predict your project outcomes?

13 What would you do, as a program manager? Dependencies Two projects in your program have struggled to get sufficient time with business Subject Matter Experts (SMEs), resulting in deployment delays These same SMEs will be needed for the upcoming project Extend the schedule of the upcoming project Find or train other SMEs that can help Look for ways to reduce your dependency on the SMEs All of the above

14 Discuss the Predicting Metrics Good Bad

15 Metric Dangers Objective Misdirection Example: First-Pass Defect Rate Misdirection: Could cause overly cautious programming / unit testing efforts, resulting in longer cycle times and unhappy customers Potential Side-effects Example: Cost Growth Side-effect: could drive projects to overestimate Example: Cost Deviation Side-effect: could drive projects to overestimate and then spend up These metrics are NOT inherently bad, they suffer from how we choose to use them! Unclear Trade-offs Example: Customer Satisfaction Trade-off: Projects with high customer satisfaction average over 25% longer cycle times

16 How to Use a Metric Confirm that the data is useful and sufficient Make sure that the data is accurate, complete, and relatively unbiased Determine what the metric is telling you Regardless of how it was designed Look for conflicting data or metrics Corporate culture plays a big role in metric usefulness Make sure that the metric makes sense in the larger context Be aware of the metrics side-effects In addition, address the culture impact of the use of the metric

17 Two Key Categories of Metrics Leading Tend to measure practices and behaviors Can be used to do real-time course corrections Tend to require more data Tend to be within the project team s ability to assess Lagging Measure outcomes Enables over-time or after-the-fact course corrections Can require data systems to implement Often part of management reports and dashboards

18 Both Leading and Lagging Indicators Can Explain Variance What is the number one one PREDICTOR driver of of schedule slippage? Team Member Turnover Quality of Risk Management Quality of Requirements Quality of Schedule Planning

19 However More Information and More Metrics Are Not Necessarily the Answer!

20 How to Design a Useful Metric Can we use it as a predictor? If Yes, be clear on what it predicts If No, have a transparent plan to use the info This is the most important step in metric development What is the metric s objective? Try to design metrics that address the key drivers (i.e., root causes) What are the potential dangers that this metric might carry? Investigate how this metric might affect other key objectives and behaviors Keep organizational culture in mind ask how this metric makes people feel try to address concerns If Yes, then make sure the data is governed Is the data easy to collect? If No, make sure that it is worth it and try to reduce the data collection overhead Be very open about how the data will be used and where it will be reported How will we use the metric? Be vigilant about enforcing the right behaviors to support the culture of the metric

21 Helpful Leading Indicators that Are Great Predictors Indicator Completeness and Quality of Work Done in Conceptual and Planning Phases Influences Nearly every outcome metric Team Health Cost Growth, Schedule Competitiveness Project Risk Score Cost Growth, Schedule Slippage, Schedule Competitiveness Degree of Technical Risk Benefit Realization, Cost Growth, Schedule Slippage Business Case Quality Benefits Realization

22 Important Elements for Monitoring Baseline/Current Cost versus Forecast Cost Baseline/Current Schedule versus Forecast Schedule Including milestones and any interim dates Team health Based on trending metrics Risks and Issues As well as a trending risk scores Including technical challenges Scope or benefit changes Resource conflicts or turnover Gate / Phase Transition Readiness Information Other lagging indicator information

23 Important Steps toward Improvement The element that is most important for improvement efforts is The question Why? Useful information can be gleaned if you look across lessons learned For leaders, take the leading indicators and lagging indicators and conduct an assessment on their drivers Key is to get to the root cause Get rid of false inferences Understand interactions Develop a plan that you can implement Determine if you have gaps in your information and plan how to address them

24 Summary Metrics serve three key purposes: monitor, improve, predict The best metrics are well designed and, often, are not the easiest to implement, necessarily Metrics should be designed to support your overall objectives and to reduce trade-offs / sideeffects Leading metrics are the most useful metrics and can be developed and used directly by project teams Lagging metrics require research in order to extract the right learnings Metric learnings can lead to process changes, training, communication, and many other actions focused on organizational, process and personal improvement

25 Questions? Law & Order is a copyright of NBCUniversal Media, LLC