The Mystery Behind Project Management Metrics Reed Shell Blue Hippo Consulting
Presentation Take-Aways Two Tools for gathering and producing metrics 10 Step Process Goal/Question/Metric Deliverable Exercises to find project metrics Use effective metrics to improve project performance Understand Evaluate Predict Control 2
Presentation Roadmap 3
The Mystery Behind Project Management Metrics Mystery 1 Where to Start 10 Step Process Mystery 2 Measuring the Right Goal Goal Question Metric(GQM) Mystery 3 Agile/Scrum Metrics Apply GQM to Scrum 4
Known Knowns Actual Work and budget Risk and reserve Discovery and Questions 5
Communicate to Senior Management and Peers Educate people as to current state Demonstrate value to the business Forecast what is needed to be done to improve the environment before a problem happens Objectives Of Metrics 6
Effective Metrics Provide Objective information to help: Make day-to-day decisions Identify project issues Correct existing problems Manage risks Evaluate performance levels Assess the impact of changes Accurately estimate and track effort, costs, and schedules Improve process, products, and services 7
Metrics Used at Different Levels Track Performance against industry benchmarks Relative growth of organization Sequential growth of organization Organizational Level Output New business goals Revised organizational norms Track Process Performance of department Efficiency of processes Customer satisfaction levels Business Unit / Vertical Level Output Process improvement Improvement of operations and project management processes Project progress against schedule Costs incurred against budget Adherence to quality standards Track Project Level Output Tactical decisions to improve health of the project 8
Where are metrics found? Project cost and effort ROI and cost of operations Performance evaluation Quality control and evaluation Reliability predictions Specific measures and algorithms Metrics 9
What is Data? A collection of facts. Portland is 307 miles away. What is Information? Meaningful data. Information is an inference drawn based on facts that data represents It s a long drive to Portland is information inferred from the data that Portland is 307 miles away. What are Metrics? Metrics establish the relationship between individual sets of data. At a speed of 75 mph, it will take roughly 4 hours to drive 307 miles. 10
Without the right information, you are just another person with an opinion. Measurements 11
To Understand To Evaluate To Predict Why Measure? To Control Image Source: Microsoft Clip Art 12
To Understand Processes, products, and resources Understand Control To Evaluate Determine status with respect to plans To Predict Make predictions and deliver Processes, Products, and Services To Control Road blocks, root causes, inefficiencies, and quality Evaluate Predict 13
Understand Gather metrics to learn about our current products, processes, and capabilities Understand Control Processes, Products, and Services Evaluate Predict 14
Evaluate Examine and analyze metrics information as a part of decision making Understand Control Processes, Products, and Services Evaluate Predict 15
Control Use metrics to control processes and product quality Understand Control Processes, Products, and Services Evaluate Predict 16
Predict Use metrics to estimate future values Understand Control Processes, Products, and Services Evaluate Predict 17
Effective Metrics Establish specific goals Establish plans?? Quality Habit Clear responsibility for goals Base rewards on results 18
Measurement Models SEI SW - CMM Software Engineering Institute (SEI) - Capability Maturity Model (CMM) Purpose is to develop and refine an organization s processes CMMI CMMI Capability Maturity Model Integration IFPUG International Function Point Users Group (IFPUG) Standardized function point counting criteria Certified Function Point Analyst The Balanced Scorecard A report used by managers to track the execution and monitor the consequences of activities OPM3 A project management best practices score 19
The Balanced Scorecard Financial Perspective How do we look to our shareholders? Customer Perspective How do our customers see us? The Balanced Scorecard Internal Business Processes Perspective At what must we excel? Learning and Growth Perspective How can we continue to improve? 20
Strategy of Metrics Practices Org Level Strategy Policy Decisions Measured by Business Results Strategy Plan Industry Benchmark Customer Satisfaction Process Compliance Tools Effectiveness Business Unit Level Manage Improve Measured by Process Capability Process Improvement Resource Utilization Customer Satisfaction Process Compliance Tools Effectiveness Project Level Tactical Execution Measured by Resources and Cost Schedule and Progress Risk Management Quality Customer Satisfaction 21
Mystery 1: Where to Start 10 Step Process 22
10 Steps for Effective Metrics 1. Identify Project Metrics Customer 2. Identify Project Goals 3. Define Project Metrics 4. Identify Project Data to Collect 5. Define Project Data Collection Methods Information and approach adapted from The 12 Steps to Useful Software Metrics by Linda Westfall The Westfall Team 6. Project Data Analysis and a Metrics Database 7. Define Project Reporting and Feedback Mechanism 8. Document the Project Metrics Process 9. Human Aspects of Project Metrics 10. Continuously improve the Process 23
1. Identify Project Metrics Customer 24
1. Identify Project Metrics Customer Two Ways to Collect Data Data Collection Method Collect Data on Everything Random Selection of Metrics Challenge Metrics are expensive Too many possible measures Easy to drown in the enormity of the task Incomplete Inconsistent Inadequate 25
Take 1 Minute to list your metrics customers 1. 2. 3. 4. 5. 26
2. Identify Project Goals GQM Goal/Question/Metrics Method 1. Identify Project goal 2. Identify questions that need answering to determine if the goal is being met 3. Identify metrics that provide necessary information to answer the questions 27
3. Define Project Metrics When Selecting Metrics: Be practical, realistic, pragmatic Start with the possible Do you have the data? 28
3. Define Project Metrics Metrics Requirements Statement The purpose of the metric is understand To evaluate the attribute in order to control predict of the entity goal(s) number by To evaluate the status of in order to the known defects Ensure all known defects are corrected before next release 29
Define Metrics Metrics Requirements Statement The purpose of the metric is understand To evaluate the attribute in order to control predict of the entity goal(s) Customer To Understand the satisfaction in order to Deliver the best product 30
Take 2 Minutes to create a metrics statement The purpose of the metric is understand To evaluate the attribute in order to control predict of the entity goal(s) To the in order to 31
4. Identify Project Data to Collect Establish Counting Criteria Define rules for mapping numbers and symbols Interpret the measure the same way Types of counting criteria Standardized units of measure (hours, days, $) Counts of items with certain characteristics Check sheets 32
5. Define Project Data Collection Methods What data to collect? Who should collect it? How to collect the data? Inaccurate Data If we wait for 100% data accuracy we may never report anything Incomplete Data Reporting incomplete data creates problems 33
6. Project Data Analysis Challenges Establishing criteria for metrics decision based on 1. Thresholds 2. Variances 3. Control limits Establishing entry/exit criteria for making decisions Criteria for Metrics Decision Thresholds, targets, or patterns should be used to o Determine the need for action or further investigation o Describe the level of confidence in a given result Provide guidance for interpreting the metrics results 34
7. Define Project Reporting & Feedback Mechanism Open Fixed Resolved Sep 11 20 21 13 Oct 11 13 24 31 Nov 11 31 22 27 Dec 11 13 18 31 Data Table Compact and structured Many details in small area Only values must be evaluated Sales 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr Pie Chart How total breaks down into parts 6 4 2 Line Graph Trends over time 0 Sep '11 Oct '11 Nov '11 Dec '11 35
7. Define Reporting & Feedback Mechanism 4 3 2 1 Bar Chart Show numbers, ratios, or proportions Quantitative or discrete variables 0 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 4 3 2 1 0 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr Grouped Bar Chart Comparison of two or more categories Easy to pick out difference Challenges 10 8 6 4 2 0 Sep '11 Oct '11 Nov '11 Dec '11 Stacked Bar Chart How total Breaks into parts Height of bar corresponds to frequency Importance of report format Timing Methods of Delivery 36
8. Document the Project Metrics Process Documenting the metrics process steps Process maps Identify metrics Standard definition Data collection methods Analysis methods and decision criteria Reporting and feedback mechanism 37
9. Human Aspects of Project Metrics The People side of the Metrics Equation 1. How measures affect people 2. How people affect measures Tell me how I will be measured, I ll tell you how I will behave. - David Packard Tell me how I will be measure, I ll tell you how I will behave. - David Packard Image Source: Microsoft Clip Art 38
9. Human Aspects of Project Metrics Human Factors: Do s and Don ts Do s Don ts Select Metric based on goals Measure individuals Provide feedback Use metrics as a stick Focus on processes, products, and services Use only one metric Obtain buy in Ignore the data 39
10. Metrics in Process Improvements Project Process Improvement Metrics Identify areas for continuous improvement Monitor the impact of those improvements 40
Mystery 2: Measuring the Right Goal Goal / Question / Metric 41
Identify Project Goals Examples Goal Possible Questions What is our current level of customer satisfaction? Maintain a high level of customer satisfaction What attribute of our products and services are most important to our customers? How do we compare with our competition? How do problems with our services affect our customers? Image Source: Microsoft Clip Art 42
Goal / Question / Metric Identify Goals Take 2 minutes to Identify a goal and list possible questions Goal Possible Questions 43
Goal Questions Goal Question Selected Metrics Ensure all known defects are corrected before release How many defects were discovered during testing Number of tickets discovered Defects arrival rate by status 44
Goal / Question / Metric Identify Metrics Take 2 minutes to select a question and list possible metrics 45
Project Management Metrics: Planning Estimation: Effort, Cost, & Schedule Metrics Goals Questions Metrics How long will the project take? Effort Estimates Deliver product on time & within budget Cost Estimates How much will the project cost? Schedule Estimates 46
Project Management Metrics: Risk Risk Management Process Prioritized Risks Plan Risk Management Plan Analyze Information Containment Actions Status Risks Identify New/Closed Risks Track Trigger Contingency Actions 47
Project Management Metrics: Risk Goals Questions Metrics Deliver product on time & within budget What is the probability of something going wrong? What is the impact if something does go wrong? Risk Probability Risk Loss Risk Exposure Which risk mitigation actions should be implemented? Risk Reduction Leverage 48
Project Management Metrics: Budget and Schedule Project Performance Metrics Examples of model based estimation methods: methods: Earned Valued Estimates Earned Valued vs. actuals Estimates o Size vs. actuals o Cost Size o Schedule Cost o Productivity Schedule Resource o Productivity utilization and staff turnover Resource utilization and staff turnover 49
Project Management Metrics: Budget and Schedule Goals Questions Metrics What is the Status of the project? Earned value Deliver product on time & within budget Are we meeting the budget? Cost performance index Are we on schedule? Schedule performance index Schedule Gantt 50
Project Management Metrics: Resource and Staff Tracking Goals Questions Metrics Do we have the resources we need? Resource Utilization Deliver product on time & within budget Do we have the staff we need? Staff Turnover Are we meeting our productivity projections? Productivity 51
List Out a Project Metric Goals Questions Metrics 52
Dangers Of Metrics 53
Dangers Of Metrics Metrics are rear-view-mirror oriented and are not good indicators of future issues Metrics do not consider human behavior (gaming) No metrics to reflect cross-functional processes Insufficient oversight of adherence to cost/schedule/performance baselines Metrics have short-term focus 54
Dangers Of Metrics The death of one man is a tragedy. The death of a million is a statistic Joseph Stalin It is one thing for numbers to represent money or products. But when big numbers start representing human beings, as Stalin told us, our ability to empathize starts to falter.but a decision using a spreadsheet to lay off four thousand people at some large corporation loses tangibility and becomes something that just needs to be done to meet certain goals. Simon Sinek Leaders Eat Last 55
Dangers Of Metrics If a metric does not have a customer it should not be produced. spending all our time reporting on the nothing we are doing because we are spending all our time reporting 56
Mystery 3: Measuring Agile/Scrum 57
Principles Behind Agile Manifesto Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Continuous attention to technical excellence and good design enhances agility. Simplicity the art of maximizing the amount of work not done is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. http://agilemanifesto.org/principles.html ReedShell@Yahoo.com 58 58
1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Key aspects: Customer satisfaction, early and continuous delivery, and delivery of software that is considered valuable by the customer Suggested metrics and measures: Return on investment (ROI) Agile earned value analysis Release milestone analysis (planned versus achieved) Customer satisfaction survey/scorecard Net Promoter Score Product Owner Feedback 59
1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Goals Questions Metrics Customer Satisfaction How Satisfied is the customer? % customers requesting support Buying and using software Product Owner Satisfaction How will Product Owner be satisfied? Net Promoter Score Early and Continuous Delivery Are we working on the highest business value? Release milestone analysis (planned versus achieved) 60
Typical Challenges Collection of data versus results Don t have time or resources My organization is changing too fast Each project is unique Teams that don t want measurement need metrics Maturity of the Organization Inaccurate data Spending too much time producing metrics that results suffer 61
Success Factors Standardize the What, Why, and How of metrics. Metrics dashboards. Start somewhere Remember soft factors and intangibles Communication tool for additional clarity 62
Last Words 1. Use metrics in understanding, controlling, evaluating or predicting your project results. 2. The goal is improvement through measurement, analysis, feedback and actions not measurement and metrics collection alone. 63
Thank you! Name Reed Shell Web Bluehippoconsulting.com Email Reed.Shell@Bluehippoconsulting.com https://www.linkedin.com/in/reedshell http://www.projectmanagement.com/profile/reedshell 64