The Mystery Behind Project Management Metrics. Reed Shell Blue Hippo Consulting

Size: px

Start display at page:

Download "The Mystery Behind Project Management Metrics. Reed Shell Blue Hippo Consulting"

Margery Priscilla Robinson
5 years ago
Views:

1 The Mystery Behind Project Management Metrics Reed Shell Blue Hippo Consulting

2 Presentation Take-Aways Two Tools for gathering and producing metrics 10 Step Process Goal/Question/Metric Deliverable Exercises to find project metrics Use effective metrics to improve project performance Understand Evaluate Predict Control 2

3 Presentation Roadmap 3

4 The Mystery Behind Project Management Metrics Mystery 1 Where to Start 10 Step Process Mystery 2 Measuring the Right Goal Goal Question Metric(GQM) Mystery 3 Agile/Scrum Metrics Apply GQM to Scrum 4

5 Known Knowns Actual Work and budget Risk and reserve Discovery and Questions 5

6 Communicate to Senior Management and Peers Educate people as to current state Demonstrate value to the business Forecast what is needed to be done to improve the environment before a problem happens Objectives Of Metrics 6

7 Effective Metrics Provide Objective information to help: Make day-to-day decisions Identify project issues Correct existing problems Manage risks Evaluate performance levels Assess the impact of changes Accurately estimate and track effort, costs, and schedules Improve process, products, and services 7

8 Metrics Used at Different Levels Track Performance against industry benchmarks Relative growth of organization Sequential growth of organization Organizational Level Output New business goals Revised organizational norms Track Process Performance of department Efficiency of processes Customer satisfaction levels Business Unit / Vertical Level Output Process improvement Improvement of operations and project management processes Project progress against schedule Costs incurred against budget Adherence to quality standards Track Project Level Output Tactical decisions to improve health of the project 8

9 Where are metrics found? Project cost and effort ROI and cost of operations Performance evaluation Quality control and evaluation Reliability predictions Specific measures and algorithms Metrics 9

10 What is Data? A collection of facts. Portland is 307 miles away. What is Information? Meaningful data. Information is an inference drawn based on facts that data represents It s a long drive to Portland is information inferred from the data that Portland is 307 miles away. What are Metrics? Metrics establish the relationship between individual sets of data. At a speed of 75 mph, it will take roughly 4 hours to drive 307 miles. 10

11 Without the right information, you are just another person with an opinion. Measurements 11

12 To Understand To Evaluate To Predict Why Measure? To Control Image Source: Microsoft Clip Art 12

13 To Understand Processes, products, and resources Understand Control To Evaluate Determine status with respect to plans To Predict Make predictions and deliver Processes, Products, and Services To Control Road blocks, root causes, inefficiencies, and quality Evaluate Predict 13

14 Understand Gather metrics to learn about our current products, processes, and capabilities Understand Control Processes, Products, and Services Evaluate Predict 14

15 Evaluate Examine and analyze metrics information as a part of decision making Understand Control Processes, Products, and Services Evaluate Predict 15

16 Control Use metrics to control processes and product quality Understand Control Processes, Products, and Services Evaluate Predict 16

17 Predict Use metrics to estimate future values Understand Control Processes, Products, and Services Evaluate Predict 17

18 Effective Metrics Establish specific goals Establish plans?? Quality Habit Clear responsibility for goals Base rewards on results 18

19 Measurement Models SEI SW - CMM Software Engineering Institute (SEI) - Capability Maturity Model (CMM) Purpose is to develop and refine an organization s processes CMMI CMMI Capability Maturity Model Integration IFPUG International Function Point Users Group (IFPUG) Standardized function point counting criteria Certified Function Point Analyst The Balanced Scorecard A report used by managers to track the execution and monitor the consequences of activities OPM3 A project management best practices score 19

20 The Balanced Scorecard Financial Perspective How do we look to our shareholders? Customer Perspective How do our customers see us? The Balanced Scorecard Internal Business Processes Perspective At what must we excel? Learning and Growth Perspective How can we continue to improve? 20

21 Strategy of Metrics Practices Org Level Strategy Policy Decisions Measured by Business Results Strategy Plan Industry Benchmark Customer Satisfaction Process Compliance Tools Effectiveness Business Unit Level Manage Improve Measured by Process Capability Process Improvement Resource Utilization Customer Satisfaction Process Compliance Tools Effectiveness Project Level Tactical Execution Measured by Resources and Cost Schedule and Progress Risk Management Quality Customer Satisfaction 21

22 Mystery 1: Where to Start 10 Step Process 22

23 10 Steps for Effective Metrics 1. Identify Project Metrics Customer 2. Identify Project Goals 3. Define Project Metrics 4. Identify Project Data to Collect 5. Define Project Data Collection Methods Information and approach adapted from The 12 Steps to Useful Software Metrics by Linda Westfall The Westfall Team 6. Project Data Analysis and a Metrics Database 7. Define Project Reporting and Feedback Mechanism 8. Document the Project Metrics Process 9. Human Aspects of Project Metrics 10. Continuously improve the Process 23

24 1. Identify Project Metrics Customer 24

25 1. Identify Project Metrics Customer Two Ways to Collect Data Data Collection Method Collect Data on Everything Random Selection of Metrics Challenge Metrics are expensive Too many possible measures Easy to drown in the enormity of the task Incomplete Inconsistent Inadequate 25

26 Take 1 Minute to list your metrics customers

27 2. Identify Project Goals GQM Goal/Question/Metrics Method 1. Identify Project goal 2. Identify questions that need answering to determine if the goal is being met 3. Identify metrics that provide necessary information to answer the questions 27

28 3. Define Project Metrics When Selecting Metrics: Be practical, realistic, pragmatic Start with the possible Do you have the data? 28

29 3. Define Project Metrics Metrics Requirements Statement The purpose of the metric is understand To evaluate the attribute in order to control predict of the entity goal(s) number by To evaluate the status of in order to the known defects Ensure all known defects are corrected before next release 29

30 Define Metrics Metrics Requirements Statement The purpose of the metric is understand To evaluate the attribute in order to control predict of the entity goal(s) Customer To Understand the satisfaction in order to Deliver the best product 30

31 Take 2 Minutes to create a metrics statement The purpose of the metric is understand To evaluate the attribute in order to control predict of the entity goal(s) To the in order to 31

32 4. Identify Project Data to Collect Establish Counting Criteria Define rules for mapping numbers and symbols Interpret the measure the same way Types of counting criteria Standardized units of measure (hours, days, $) Counts of items with certain characteristics Check sheets 32

33 5. Define Project Data Collection Methods What data to collect? Who should collect it? How to collect the data? Inaccurate Data If we wait for 100% data accuracy we may never report anything Incomplete Data Reporting incomplete data creates problems 33

34 6. Project Data Analysis Challenges Establishing criteria for metrics decision based on 1. Thresholds 2. Variances 3. Control limits Establishing entry/exit criteria for making decisions Criteria for Metrics Decision Thresholds, targets, or patterns should be used to o Determine the need for action or further investigation o Describe the level of confidence in a given result Provide guidance for interpreting the metrics results 34

7. Define Project Reporting & Feedback Mechanism Open Fixed Resolved Sep 11 20 21 13 Oct 11 13 24 31 Nov 11 31 22 27 Dec 11 13 18 31 Data Table Compact and structured Many details in

35 7. Define Project Reporting & Feedback Mechanism Open Fixed Resolved Sep Oct Nov Dec Data Table Compact and structured Many details in small area Only values must be evaluated Sales 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr Pie Chart How total breaks down into parts Line Graph Trends over time 0 Sep '11 Oct '11 Nov '11 Dec '11 35

7. Define Reporting & Feedback Mechanism 4 3 2 1 Bar Chart Show numbers, ratios, or proportions Quantitative or discrete

or more categories Easy to pick out difference Challenges 10 8 6 4 2 0 Sep '11 Oct '11 Nov '11 Dec '11 Stacked Bar Chart

36 7. Define Reporting & Feedback Mechanism Bar Chart Show numbers, ratios, or proportions Quantitative or discrete variables 0 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr st Qtr 2nd Qtr 3rd Qtr 4th Qtr Grouped Bar Chart Comparison of two or more categories Easy to pick out difference Challenges Sep '11 Oct '11 Nov '11 Dec '11 Stacked Bar Chart How total Breaks into parts Height of bar corresponds to frequency Importance of report format Timing Methods of Delivery 36

37 8. Document the Project Metrics Process Documenting the metrics process steps Process maps Identify metrics Standard definition Data collection methods Analysis methods and decision criteria Reporting and feedback mechanism 37

How people affect measures Tell me how I will be measured, I ll tell you how I

38 9. Human Aspects of Project Metrics The People side of the Metrics Equation 1. How measures affect people 2. How people affect measures Tell me how I will be measured, I ll tell you how I will behave. - David Packard Tell me how I will be measure, I ll tell you how I will behave. - David Packard Image Source: Microsoft Clip Art 38

39 9. Human Aspects of Project Metrics Human Factors: Do s and Don ts Do s Don ts Select Metric based on goals Measure individuals Provide feedback Use metrics as a stick Focus on processes, products, and services Use only one metric Obtain buy in Ignore the data 39

40 10. Metrics in Process Improvements Project Process Improvement Metrics Identify areas for continuous improvement Monitor the impact of those improvements 40

41 Mystery 2: Measuring the Right Goal Goal / Question / Metric 41

42 Identify Project Goals Examples Goal Possible Questions What is our current level of customer satisfaction? Maintain a high level of customer satisfaction What attribute of our products and services are most important to our customers? How do we compare with our competition? How do problems with our services affect our customers? Image Source: Microsoft Clip Art 42

43 Goal / Question / Metric Identify Goals Take 2 minutes to Identify a goal and list possible questions Goal Possible Questions 43

44 Goal Questions Goal Question Selected Metrics Ensure all known defects are corrected before release How many defects were discovered during testing Number of tickets discovered Defects arrival rate by status 44

45 Goal / Question / Metric Identify Metrics Take 2 minutes to select a question and list possible metrics 45

46 Project Management Metrics: Planning Estimation: Effort, Cost, & Schedule Metrics Goals Questions Metrics How long will the project take? Effort Estimates Deliver product on time & within budget Cost Estimates How much will the project cost? Schedule Estimates 46

47 Project Management Metrics: Risk Risk Management Process Prioritized Risks Plan Risk Management Plan Analyze Information Containment Actions Status Risks Identify New/Closed Risks Track Trigger Contingency Actions 47

48 Project Management Metrics: Risk Goals Questions Metrics Deliver product on time & within budget What is the probability of something going wrong? What is the impact if something does go wrong? Risk Probability Risk Loss Risk Exposure Which risk mitigation actions should be implemented? Risk Reduction Leverage 48

Project Management Metrics: Budget and Schedule Project

methods: methods: Earned Valued Estimates Earned Valued vs.

actuals o Cost Size o Schedule Cost o Productivity Schedule

49 Project Management Metrics: Budget and Schedule Project Performance Metrics Examples of model based estimation methods: methods: Earned Valued Estimates Earned Valued vs. actuals Estimates o Size vs. actuals o Cost Size o Schedule Cost o Productivity Schedule Resource o Productivity utilization and staff turnover Resource utilization and staff turnover 49

50 Project Management Metrics: Budget and Schedule Goals Questions Metrics What is the Status of the project? Earned value Deliver product on time & within budget Are we meeting the budget? Cost performance index Are we on schedule? Schedule performance index Schedule Gantt 50

51 Project Management Metrics: Resource and Staff Tracking Goals Questions Metrics Do we have the resources we need? Resource Utilization Deliver product on time & within budget Do we have the staff we need? Staff Turnover Are we meeting our productivity projections? Productivity 51

52 List Out a Project Metric Goals Questions Metrics 52

53 Dangers Of Metrics 53

54 Dangers Of Metrics Metrics are rear-view-mirror oriented and are not good indicators of future issues Metrics do not consider human behavior (gaming) No metrics to reflect cross-functional processes Insufficient oversight of adherence to cost/schedule/performance baselines Metrics have short-term focus 54

But when big numbers start representing human beings, as Stalin told us, our ability to empathize starts to falter.

55 Dangers Of Metrics The death of one man is a tragedy. The death of a million is a statistic Joseph Stalin It is one thing for numbers to represent money or products. But when big numbers start representing human beings, as Stalin told us, our ability to empathize starts to falter.but a decision using a spreadsheet to lay off four thousand people at some large corporation loses tangibility and becomes something that just needs to be done to meet certain goals. Simon Sinek Leaders Eat Last 55

56 Dangers Of Metrics If a metric does not have a customer it should not be produced. spending all our time reporting on the nothing we are doing because we are spending all our time reporting 56

57 Mystery 3: Measuring Agile/Scrum 57

Principles Behind Agile Manifesto Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development.

Business people and developers must work together daily throughout the project. Build projects around motivated individuals.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Working software is the primary measure of progress.

58 Principles Behind Agile Manifesto Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Continuous attention to technical excellence and good design enhances agility. Simplicity the art of maximizing the amount of work not done is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. ReedShell@Yahoo.com 58 58

59 1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Key aspects: Customer satisfaction, early and continuous delivery, and delivery of software that is considered valuable by the customer Suggested metrics and measures: Return on investment (ROI) Agile earned value analysis Release milestone analysis (planned versus achieved) Customer satisfaction survey/scorecard Net Promoter Score Product Owner Feedback 59

60 1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Goals Questions Metrics Customer Satisfaction How Satisfied is the customer? % customers requesting support Buying and using software Product Owner Satisfaction How will Product Owner be satisfied? Net Promoter Score Early and Continuous Delivery Are we working on the highest business value? Release milestone analysis (planned versus achieved) 60

Typical Challenges Collection of data versus results Don t have time or resources My organization is changing too fast Each project is unique Teams

61 Typical Challenges Collection of data versus results Don t have time or resources My organization is changing too fast Each project is unique Teams that don t want measurement need metrics Maturity of the Organization Inaccurate data Spending too much time producing metrics that results suffer 61

62 Success Factors Standardize the What, Why, and How of metrics. Metrics dashboards. Start somewhere Remember soft factors and intangibles Communication tool for additional clarity 62

63 Last Words 1. Use metrics in understanding, controlling, evaluating or predicting your project results. 2. The goal is improvement through measurement, analysis, feedback and actions not measurement and metrics collection alone. 63

Thank you! Name Reed Shell Web Bluehippoconsulting.com Email Reed.Shell@Bluehippoconsulting.

64 Thank you! Name Reed Shell Web Bluehippoconsulting.com 64

ONE! TEAM! 2010, Nick Athanassiadis. All rights reserved.!

ONE! TEAM! 2010, Nick Athanassiadis. All rights reserved.! ONE! TEAM! 1! ONE TEAM! ONE MISSION! 2! CLIENTS! 3! WHATʼS IN OUR! BOX?! 4! 5! POST-IT!!!!!!and! 6! using! AGILE & SCRUM! METHODOLOGIES! 7! AGILE MANIFESTO! 8! AGILE MANIFESTO! Our highest priority is