Determining what to measure. Determining how to measure. Software measure classification Goal-based paradigms: Applications of GQM and GQIM

Transcription

1 Contents SENG 421: Software Metrics Goal-Based Software Measurement Framework (Chapter 3) Department of Electrical & Computer Engineering, University of Calgary B.H. Far Software measure classification Goal-based paradigms: Goal-Question-Metrics (GQM) Goal-Question-Indicator-Metrics (GQIM) Applications of GQM and GQIM 2 Goal-Based Measurement (GBM) The primary question in goal-based measurement: What do we want to know or learn? instead of What metrics should we use? Because the answers depend on your goals, no fixed set of metrics is universally appropriate. Instead of attempting to develop general-purpose measures, one has to describe an adaptable process that users can use to identify and define measures that provide insights into their own development problem. GBM Process Determining what to measure Identifying entities Classifying entries to be examined Determining relevant goals Determining how to measure Inquire about metrics Assign metrics 3 4 Identifying Entities Product, process, resource [Fenton 91]. Artifacts, activities, agents [Armitage 94]. Both schemes are incomplete and haven t enough representational power. E.g., neither seems to deal with entities such as defects. Is a defect a product, a process, or a resource? Is defect an artifact, activity, or agent? Process, Product, Resource Process A collection of software related activities usually associated with some timescale. Different SE processes: development, maintenance, testing, reuse, configuration and management process, etc. Product Any artifacts, deliverables or documents that result from a process activity. Resource Entities required by a process activity. far@ucalgary.ca 5 far@ucalgary.ca 6

2 Types of Attributes Internal attributes Attributes that can be measured entirely in terms of the process, product or resource itself separate from its behaviour e.g., size External attributes Attributes that can be measured with respect to how the process, product or resource relates to its environment through its behaviour e.g., quality Sample Process Measures /1 Process Entities Attributes Possible Measures development elapsed time Calendar days, working days process milestones Calendar dates development effort Staff-hours, days, or months phase containment Percent of total defects found in phase where introduced process compliance Percent of tasks complying with standard procedures or directives performance Number of tests passed divided by number of tests executed test process volume Number of tests scheduled progress Number of tests executed Number of tests passed 7 8 Sample Process Measures /2 Process Entities Attributes Possible Measures detailed design elapsed time design quality calendar days, working days defect density: number of design defects found in down-stream activities divided by a measure of product size, such as function points or physical source lines of code. maintenance cost dollars per year staff-hours per change request Change request backlog size number of change requests awaiting service estimated effort (staff-hours) for pending requests Sample Product Measures /1 Product Entities Attributes Possible Measures system size number of modules number of bubbles in a data-flow diagram number of function points number of physical source lines of code number of memory bytes or words required (or allocated) defect density defects per KLOC defects per function point module length physical source lines of code logical source statements percent reused ratio of unchanged physical lines to total physical lines, comments and blanks excluded far@ucalgary.ca 9 far@ucalgary.ca 10 Sample Product Measures /2 Product Entities Attributes Possible Measures unit number of linearly McCabe s complexity independent flowpaths document length number of pages line of code statement type type names how produced name of production method programming language language name defect type type names origin name of activity where introduced severity an ordered set of severity classes effort to fix staff-hours Sample Resource Measures Resource Entities Attributes Possible Measures Assigned staff team size number of people assigned experience years of domain experience years of programming experience CASE tools type name of type Is used? yes/no (a binary classification) time start date, due date calendar dates elapsed time days Execution time CPU clocks far@ucalgary.ca 11 far@ucalgary.ca 12

3 GQM Approach /1 Goal-Question-Metric (GQM) approach to process metrics provides a framework for deriving measures from organization or business goals. References: V.R. Basili and D. Weiss (1984), A Methodology for Collecting Valid Software Engineering Data, IEEE Trans. Software Engineering, vol. 10, pp V.R. Basili and H.D. Rombach (1988), The Tame Project: Towards Improvement-Oriented Software Environments, IEEE Trans. Software Engineering, vol.14, pp Basili s GQM Basili s GQM process model [Basili 1992] Phase Description 1 Developing a set of corporate, division and project goals 2 Generating questions that define those goals as completely as possible in a quantifiable way Specifying measures (metrics) needed to be collected to answer those 3 questions and to track process and product conformance to the goals 4 Developing mechanisms for data collection 5 Collecting, validating and analyzing the data in real time to provide feedback to projects for corrective action, to assess conformance to the goals and make recommendations for future improvements far@ucalgary.ca 13 far@ucalgary.ca 14 STTI-KL s GQM GQM process model defined by the Software Technology Transfer Initiative - Kaiserslautern (STTI-KL) in 1995 [Gresse 1995]. No. Phase Description Prestudy Identification of the GQM goals Production of GQM plan The collection of information which is relevant to the introduction of a GQM-based measurement program Based on the prestudy, the GQM goals are derived from the informal organizational improvement goals and project goals Finding out the implicit knowledge of people with respect to the measurement goal by interviewing; Merge the results; Refine a GQM plan; Review the GQM plan far@ucalgary.ca 15 STTI-KL s GQM (cont d) No. Phase Description Production of the measurement plan Collection and validation of the data Analysis of the data Packaging the experiences Defining the data collection procedures and checking whether the prescribed data collection procedures are consistent with the project plan Collecting measurement data according to the defined procedures; validated and stored data Analyzing and interpreting of the collected data by feedback sessions Recording the results and experiences gained by the measurement program far@ucalgary.ca 16 Solingen s GQM GQM process model defined by Solingen [Solingen et al. 1999] No. Phase Procedures 1 Planning 2 Definition Establish GQM team; Select improvement area; Select application project and establish a project team; Create project plan; Training and promotion Define measurement goals; Review or produce software process models; Conduct GQM interviews; Define questions and hypotheses; Review questions and hypotheses; Define metrics; Check on metric consistency and completeness; Produce a GQM plan; Produce measurement plan; Produce analysis plan; Review plans Solingen s GQM (cont d) No. Phase Procedures 3 Data collection 4 Interpretation Defining the data collection steps, forms, and using tools; Data collection start up and training; Building a measurement support system (MSS) Preparation of a feedback session; Holding a feedback session; Reporting interpretations of measurement results; Cost and benefits analysis of a measurement program far@ucalgary.ca 17 far@ucalgary.ca 18

4 SEI s GQM The goal-driven process model defined by Software Engineering Institute (SEI) in1996 [Park 1996]. This GQM process model defines ten steps to fulfill the task of developing a software measurement plan. It focuses on the whole process of converting a business goal into several specific measurement goals and refining to a software measurement plan. This process model is easy to understand and follow in that it works in a pipeline style, which means that the output of a step is the input of the next step [Chen 2003]. far@ucalgary.ca 19 Comparison of Models GQM Process Model Simplicity Clearness Completeness Process model defined in by Basili 1992 X Process model defined by STTI-KL in 1995 X Process model defined by Solingen in 1999 X X Goal-driven process defined by SEI in 1996 X X X We will focus on this far@ucalgary.ca 20 GQM Approach /2 Goal: List major goals of development or maintenance project. Question: Derive from each goal the questions that must be answered to determine if the goals are being met. Metrics: Decide what must be measured in order to be able to answer the questions adequately. GQM: Example /1 far@ucalgary.ca 21 Example from Fenton s Book far@ucalgary.ca 22 GQM: Example /2 You are manager for a software development team and you have to decide upon the release time of your product. Construct a GQM tree related to this goal. Questions: What is reliability requirement? Size of code Number of failures discovered GQM: Example /3 Construct a GQM graph for the goal of improving maintainability of your developed software. Goal: suitable Release time What is current reliability? What are time constraints? Number of test cases Hours of use Number of person days Available for testing Days till deadline far@ucalgary.ca 23 far@ucalgary.ca 24

5 GQM: Data Analysis Suppose that you propose to collect data to examine the effects of programmer experience on development effort. You are expecting that effort (staff-hours) will decrease as the experience of the people assigned increases (Fig. a) but the collected data shows effort increasing with increasing experience (Fig. b). What may be wrong? Case Study: AT&T s GQM Goal: Better code base inspection Subgoals: Better inspection planning: [Plan] Better monitor and control of the code: [Control] Improving code inspection: [Improve] Technology factor? under-staffing? Cognitive overload? Rapidly changing requirements? staff morale? team-work problems? far@ucalgary.ca 25 far@ucalgary.ca 26 AT&T s GQM /1 GA: Code inspection: Plan Q1: How much does the inspection process cost? M1-1: Average effort per KLOC M1-2: Percentage of re-inspection Q2: How much calendar time does the inspection process take? M2-1: Average effort per KLOC M2-2: Total KLOC inspected AT&T s GQM /2 GB: Code inspection: Monitor & control Q1: What is the quality of the inspection software? M1-1: Average fault detected per KLOC M1-2: Average inspection rate M1-3: Average preparation rate Q2: To what degree did the staff conform to procedure? M2-1: Average inspection rate M2-2: Average preparation rate M2-3: Average lines of code inspected M2-4: Percentage of re-inspection Q3: What is the status of the inspection process? M3: Total KLOC inspected far@ucalgary.ca 27 far@ucalgary.ca 28 AT&T s GQM /3 GC: Code inspection: Improve Q1: How effective is the inspection process? M1-1: Defect removal efficiency M1-2: Average fault detected per KLOC M1-3: Average inspection rate M1-4: Average preparation rate M1-5: Average lines of code inspected Q2: What is the productivity of the inspection process? M2-1: Average effort per fault detected M2-2: Average inspection rate M2-3: Average preparation rate M2-4: Average lines of code inspected Case Study: HP s GQM Three goals: A: Maximize customer satisfaction B: Minimize engineering effort and schedule C: Minimize defects far@ucalgary.ca 29 far@ucalgary.ca 30

6 Example: HP s GQM-A /1 GA: Maximize customer satisfaction. QA1: What are the attributes of customer satisfaction? MA1: Functionality, usability, reliability, performance, supportability. QA2: What are the key indicators of customer satisfaction? MA2: Survey, quality function deployment (QFD). QA3: What aspects result in customer satisfaction? MA3: Survey, QFD. QA4: How satisfied are customers? MA4: Survey, interview record, number of customers severely affected by defects. QA5: How many customers are affected by a problem? MA5: Number of duplicate defects by severity Example: HP s GQM-A /2 GA: Maximize customer satisfaction. QA6: How many problems are affecting the customer? MA6-1: Incoming defect rate MA6-2: Open critical and serious defects MA6-3: Defect report/fix ratio MA6-4: Post-release defect density QA7: How long does it take to fix a problem? MA7-1: Mean time to acknowledge problem MA7-2: Mean time to deliver solution MA7-3: Scheduled vs. actual delivery MA7-4: Customer expectation of time to fix far@ucalgary.ca 31 far@ucalgary.ca 32 Example: HP s GQM-A /2 GA: Maximize customer satisfaction. QA8: How does installing a fix affect the customer? MA8-1: Time customers operation is down MA8-2: Customers effort required during installation QA9: Where are the bottlenecks? MA9: Backlog status, time spent doing different activities Example: HP s GQM-B /1 GB: Minimize engineering effort & schedule. QB1: Where are the worst rework loops in the process? MB1: Person-months by product-component-activity. QB2: What are the total life-cycle maintenance and support costs for the product? MB2-1: Person-months by product-component-activity. MB2-2: Person-months by corrective, adaptive, perfective maintenance. QB3: What development methods affect maintenance costs? MB3: Pre-release records of methods and post-release costs. far@ucalgary.ca 33 far@ucalgary.ca 34 Example: HP s GQM-B /2 GB: Minimize engineering effort & schedule. QB4: How maintainable is the product as changes occur? MB4-1: Incoming problem rate MB4-2: detect density MB4-3: Code stability MB4-4: Complexity MB4-5: Number of modules changed to fix one detect QB5: What will process monitoring cost and where are the costs distributed? MB5: Person-months and costs QB6: What will maintenance requirements be? MB6-1: Code stability, complexity, size MB6-2: Pre-release defect density Example: HP s GQM-B /3 GB: Minimize engineering effort & schedule. QB7: How can we predict cycle time, reliability, and effort? MB7-1: Calendar time MB7-2: Person-month MB7-3: Defect density MB7-4: Number of detects to fix MB7-5: Defect report/fix ratio MB7-6: Code stability MB7-7: Complexity MB7-8: Number of lines to change far@ucalgary.ca 35 far@ucalgary.ca 36

7 Example: HP s GQM-B /4 GB: Minimize engineering effort & schedule. QB8: What practices yield best results? MB8: Correlations between pre-release practices and customer satisfaction data QB9: How much do the maintenance phase activities cost? MB9: Personal-months and cost QB10: What are major cost components? MB10: Person-months by product-component-activity QB11: How do costs change over time? MB11: Track cost components over entity maintenance lifecycle. Example: HP s GQM-C /1 GC: Minimize defects QC1: What are key Indicators of process health and how are we doing? MC1: Release schedule met, trends of defect density, serious and critical detects. QC2: What are high-leverage opportunities for preventive maintenance? MC2-1: Defect categorization MC2-2: Code stability QC3: Are fixes effective with less side effects? MC3: Defect report/fix ratio far@ucalgary.ca 37 far@ucalgary.ca 38 Example: HP s GQM-C /2 GC: Minimize defects QC4: What is the post-release quality of each module? MC4: Defect density, critical and serious detects. QC5: What are we doing right? MC5-1: Defect removal efficiency MC5-2: Defect report/fix ratio QC6: How do we know when to release? MC6-1: Predicted defect detection and remaining defects. MC6-2: Test coverage. Example: HP s GQM-C /3 GC: Minimize defects QC7: How effective is the development process in preventing defects? MC7: Post-release detect density QC8: What can we predict will happen post-release based on pre-release data? MC8: Corrections between pre-release complexity, defect density, stability, and customer survey data. QC9: What defects are getting through and there causes? MC9: Detect categorization far@ucalgary.ca 39 far@ucalgary.ca 40 Goal Based Software Measurement: GQ(I)M Process GQM Approach: Review Goal: List major goals of development or maintenance project. Question: Derive from each goal the questions that must be answered to determine if the goals are being met. Metrics: Decide what must be measured in order to be able to answer the questions adequately. far@ucalgary.ca 42

8 GQ(I)M: Concept A methodology to convert a business goal into a measurement plan. Business Goal Business Goal: Reduce cost? GQIM Process Measurement Plan Measurement Plan: GQ(I)M: Precepts The GQ(I)M method is based on 3 precepts, and it consists of 10 steps The three precepts are: Measurement goals are derived from business goals Evolving mental models provide context and focus GQ(I)M translates informal business goals into executable measurement structures far@ucalgary.ca 43 far@ucalgary.ca 44 GQ(I)M Process GQ(I)M: Steps /1 10 steps of the GQIM process A home brewed tool to automate this process exists (ISMS) Identify Goals 1) Identify Business Goals 2) Identify what to know 3) Identify subgoals 4) Identify entities & attributes 5) Formalize Measurement Goals Define Indicators 6) Identify questions and indicators 7) Identify measures 8) Define measures Create Action Plan 9) Identify actions G Q I M BG1 MG1 MG1 MG1 MG1 Q1 Q2 Q3 I1 I2 I3 I4 M M2 M Identify business goals 2. Identify what you want to know or learn 3. Identify subgoals 4. Identify entities and attributes related to subgoals 5. Formalize measurement goals Business goal Measurement goal 10) Create plan Action Plan far@ucalgary.ca 45 far@ucalgary.ca 46 GQ(I)M: Steps /2 6. Identify quantifiable questions and indicators that will be used to achieve measurement goals 7. Identify data elements that will be collected to construct the indicators to answer questions 8. Define measures to be used, and make these definitions operational 9. Identify actions that will be taken to implement the measures 10. Prepare a plan for implementing the measures Measurement goal Measurement plan GQ(I)M: Steps /3 The GQ(I)M method begins with identifying business goals and breaking them down into manageable subgoals. It ends with a plan for implementing well-defined measures and indicators that support the goals. It can also maintain traceability back to the business goals far@ucalgary.ca 47 far@ucalgary.ca 48

9 GQ(I)M: Process /1 Processes are composed of four types of elements: Things (i.e., entities) they receive (e.g., inputs and resources) these are used or consumed Things (i.e., entities) they produce (e.g., outputs) these include products, by-products, and effects Things (i.e., entities) they consist of (e.g., activities, flowpaths, and agents) the structure of the process Things (i.e., entities) they hold or retain (e.g., internal artifacts, such as inventory and work in process) receives consists of holds Process produces 49 GQ(I)M: Process /2 Receives Resources People, facilities, tools, money Consumables Time, energy, effort Guidelines and directions Policies, procedures, goals, constraints, rules, laws, regulations, training, instructions Products and by-products from other processes 50 GQ(I)M: Process /3 Consists of Processes Tasks, steps, activities, subprocesses, transformations, reviews, inspections Controllers Flow controllers Flowpaths Product paths, resource paths, data paths, control paths Buffers and dampers Queues, stacks GQ(I)M: Process /4 Holds inventory materials work in process tools data knowledge experience GQ(I)M: Process /5 Produces Products Requirements, specifications, plans, architectures, preliminary designs, detailed designs, reviewed designs, code, reviewed code, integrated code, tested code, test cases, test results, tested components, change requests, documentation, defect reports, data By-products Knowledge, experience, skills, process improvements, waste and scrap data, defects Step 1: Identify Business Goals The business goal can be initiated at any organizational level Examples: Reduce time to market (TTM) Improve customer satisfaction Improve quality of the code 1 2 far@ucalgary.ca 53 far@ucalgary.ca 54

10 Templates for Goal Definition (1) Purpose: To (characterize, evaluate, predict, motivate, etc.) the (process, product, model, metric, etc.) in order to (understand, assess, manage, engineer, learn, improve, etc.) it. Example: Evaluating maintenance process in order to improve it. (2) Perspective: Examine the (cost, effectiveness, correctness, defects, changes, product measures, etc.) from the viewpoint of the (developer, manager, customer, etc.) Example: Examine the cost of software development from the viewpoint of the manager. Templates for Goal Definition (3) Environment: The environment consists of the following: process factors, people factors, problem factors, methods, tools, constraints, etc. Example: The maintenance staff are poorly motivated programmers who have limited access to tools Step 1: Identify Goals Example The output of Step 1 is a sorted checklist of business goals (i.e., management goals, development goals, and maintenance goals, etc.) along with their definitions. If more than one goal: generate plan for each goal separately. Typical expansion rate: 1-4 times Business Goal #1: Business Goal #2: Business Goal #3: Business Goal #n: Primary Business Goal: Saving money by improving productivity of software development team Examine the effect of productivity of software development team on project costs (Analyzed from project manager s perspective) far@ucalgary.ca 57 far@ucalgary.ca 58 Step 2: Identify What to Know Step 2: Identify What to Know Identifying what is needed to be known in order to understand, assess, predict, or improve the activities related to achieving goals by asking questions such as: What activities do we manage or execute? What do we want to achieve or improve? and by completing statements such as To do this, we will need to Repeat these questions several times and break top-level goals down into specific things should be accomplished and issues that are needed to be addressed. The output of this step is the entity-question checklist. Make sure that entities addressed should be in four categories: input and resources, products and by-products, internal artefacts, and activities and flowpaths. For each entity, list questions that can help us address those business goals. far@ucalgary.ca 59 far@ucalgary.ca 60

11 Entity-Question Checklist /1 1. Start with one of the top-level goals identified in Step Identify the persons or groups whose concerns will be addressed. (i.e., manager, developer, customer, etc.) This defines the perspective and the roles that you and the team will assume in Tasks 3 through 6 here and in the remaining steps of the process. 3. Create rough sketches of the relevant processes that you, in your role, manage or affect. As you do this, be guided by what you want to achieve and the issues you will have to address to achieve it. Entity-Question Checklist /2 4. List the important things (entities) in your processes that you, in your role, manage or influence. Make sure that you address the four kinds of process entities below: Inputs and resources Products and by-products Internal artifacts such as inventory and work in process Activities and flowpaths You may also want to list some of the environmental entities outside your processes that affect your work. far@ucalgary.ca 61 far@ucalgary.ca 62 Entity-Question Checklist /3 5. For each entity, list questions that, if answered, would help you, in your role, plan and manage progress toward your goals. For example: How big is it? How much is there? How many components? How fast is it? How long does it take? How much does it cost? Entity-Question Checklist /4 6. Step back and look at your process as a whole to see if anything is missed. By asking questions such as: Is the process stable? How is it performing now? What limits our capability? What determines quality? What determines success? What things can we control? What do our customers want? What limits our performance? What might signal early warnings? Where is backlog occurring? How big is our backlog? What could go wrong? How will we know? You may discover additional entities whose properties may be worth measuring. 7. Repeat Tasks 1 6 for your other goals. far@ucalgary.ca 63 far@ucalgary.ca 64 Entities of Interest Questions Related to Business Goal Inputs People How productive are people right now? and Resources In what areas do they need improvement? Extent of individual morale? Do we need more team members? Is our staff over-worked? Computer Aided Software Engineering Tools Are the tools sufficient or do they need to be upgraded? How do tools affect productivity? Subcontractors Would subcontracting aid our productivity? Is it worth it? Customer Requests for Change How much do changes in the project affect productivity? Entities of Interest Activities Development and Flow Paths Testing Fixing Questions Related to Business Goal Are we using the most productive development methodology? Do we effectively structure the development team? Are we spending too much time in the development phase? Are we using manual testing or automated testing? Are the tests finding enough defects? Are we spending too much time on testing? Is the response time for fixing bugs reasonable? Are high-priority changes getting implemented in a timely fashion? Are we spending too much time debugging? far@ucalgary.ca 65 far@ucalgary.ca 66

12 Step 3: Identify Subgoals /1 Entities of Interest Products Documents and Byproducts Source Code Plans Budget Questions Related to Business Goal Are the documents we produce readable? Is it possible to trace system features from one document to the next? Is the source code consistent with the documents? Is the source code error free? Does source code follow programming standards? Do the plans change too much? Do we have enough money to increase salaries and invest in tools? Step 3: Identify Subgoals /2 Grouping related questions helps identify subgoals In Step 3, you identify the questions that you have about the entities, then group them and identify the issues they address. Then groupings of issues and questions translate naturally into candidate subgoals. Sometimes, you may find several issues mapping into a single subgoal, or single issues mapping into several subgoals. far@ucalgary.ca 69 Questions Related to Development Team Productivity GROUPING #1 How productive are our employees right now? (People) In what areas do they need improvement? Extent of overall office morale? Do we need more team players? Is our staff over-worked? GROUPING #2 Are the tools sufficient or do they need to be upgraded? (Development) How do tools affect productivity? Would subcontracting aid our productivity? Is it worth it? Are we using the most productive development methodology? Are we spending too much time in the development phase? Is the source code consistent with the documents? Is the source code error free? Does source code follow programming standards? Do we have enough money to invest in tools? far@ucalgary.ca 70 Questions Related to Development Team Productivity GROUPING #3 (Response to Change) GROUPING #4 (Quality Assurance) How much do changes in the project affect productivity? Do the plans change too much? Are we using manual testing or automated testing? Are the tests finding enough defects? Are we spending too much time on testing? Is the response time for fixing bugs reasonable? Are high-priority changes getting implemented in a timely fashion? Are we spending too much time debugging? Do we need better testing to be done? Are the documents we produce readable? Is it possible to trace system features from one document to the next? Derived Subgoals (from Project Manager s Perspective) Subgoal #1 Improve performance of staff Subgoal #2 Improve code development processes Subgoal #3 Minimize the negative effects of project changes in productivity Subgoal #4 Improve quality assurance far@ucalgary.ca 71 far@ucalgary.ca 72

13 Step 4: Identify Entities & Attributes /1 4 Step 4: Identify Entities & Attributes /2 Once having a list of questions, you should examine each question and identify entities implicit in it. Then list pertinent attributes associated with each entity. Pertinent attributes are those which, if quantified, help answer your question or establish a context for interpreting the answers. Pertinent attributes are usually cited in the question, either explicitly or implicitly. List of entities and the attributes for each entity are the principal outputs of this step. The attributes will become candidates for the things that should be measured. far@ucalgary.ca 73 far@ucalgary.ca 74 Subgoal 1: Improve the performance of our staff Question 1: Currently, how productive is our development team? Entity: Development team Attributes: Team personality factors Expertise of the development organization Team s analysis and design techniques Knowledge of programming languages Subgoal 1: Improve the performance of our staff Question 2: How is the overall office s morale? Entity: Working Environment Attributes: Accommodations Incentives Extra-curricular activities Hardware and software used Work time to break time ratio Workspace (room and desk size, ventilation) far@ucalgary.ca 75 far@ucalgary.ca 76 Subgoal 2: Improve Code Development Process Question 1: Are we spending too much time in the development phase? Entity: The development process Attributes: Average duration of development process, per line of code Percentage of development process duration for average project Company expectations for percentage duration of code development Subgoal 3: Minimize the negative effects of project changes in productivity Question 1: How much do changes in the project affect productivity? Entity: Set of change requests received from the customer Attributes: Number of change requests Estimated total effort to satisfy change requests Total effort for project far@ucalgary.ca 77 far@ucalgary.ca 78

14 Subgoal 4: Improve quality assurance Question 1: Are we using manual testing or automated testing? Entity: Testing method Attributes: Type (name of type) Is used? Yes/no (a binary classification) Subgoal 4: Improve quality assurance Question 2: How effective are the tests? Are we spending too much time on testing? Entity: Testing Attributes: Volume number of tests scheduled Progress number of tests executed number of tests passed Step 5: Formalize Measurement Goals /1 5 What is a Measurement Goal? Business goals (or subgoals) are usually represented by a phrase or sentence in natural language. A measurement goal (or subgoal) is a semi-formal representation of a business goal (or subgoal), composed of 4 components: An object of interest (entity) A purpose A perspective A description of environment & constraints far@ucalgary.ca 81 far@ucalgary.ca 82 What is a Measurement Goal? Traceability: Measurement subgoals should be traced back to the subgoals and business goals to show that they are consistent with the business objective. Active & Passive Measurement Goals Active measurement goals are directed toward controlling processes or causing changes to products, processes, resources, or environments. These are the kinds of goals that are found in project management and process improvement activities. Passive measurement goals are meant to enable learning or understanding. Passive goals are often accomplished by characterizing objects of interest according to some productivity or quality model. far@ucalgary.ca 83 far@ucalgary.ca 84

15 Active & Passive Measurement Goals: Examples Active Goals Meet the scheduled completion date Reduce variability Improve product reliability Improve productivity of the process Improve time-to-market Reduce employee turnover Passive Goals Understand the current development process Identify root causes Assess product maintainability Identify capabilities and trends, so that we can better predict future performance Understand relationships among attributes, so that we can develop models for predicting and estimating Measurement Goal: Object Anything real or abstract, that we want to describe or know more about is a potential object for measurement. An object is an entity we want to describe with measured values. Object of interest (entity): a process, product, resource, task, activity, agent, artefact, metric, environment, etc. far@ucalgary.ca 85 far@ucalgary.ca 86 Measurement Goal: Purpose Purpose of a measurement activity may be to understand, predict, plan, control, compare, assess, or improve some productivity or quality aspect of the object. Examples include: cost, size, reliability, test coverage, responsiveness, peer review effectiveness, process compliance, time to market, quality, customer satisfaction. Purpose: the in order to it. characterize, analyze, evaluate, etc. <entity>, <aspect>, <attribute(s)>, etc. understand, baseline, predict, plan, control, assess, compare, improve, etc. Measurement Goal: Perspective The perspective identifies who is interested in the measurement results, such as: developer, maintainer, manager, or customer. The perspective is stated to clarify the purpose of the measurement activity. Perspective: Examine the from the point of view of (the). modifiability, quality, changes, defects, defect types, backlog, behaviour, stability, progress, <specific attribute(s)>, etc. developer, manager, customer, engineer, process improvement team, senior management, etc. far@ucalgary.ca 87 far@ucalgary.ca 88 Measurement Goal: Environment A description of environment provides a context for interpreting measurement results. The environment includes everything that affects or is affected by the object to be measured, such as time, resources, unusual performance criteria, etc., as well as constraints on the scope or time span of the measurement process itself. Environment: List or otherwise describe the environmental factors & related parameters that should be understood to put the observed results in context. Focus on describing similarities to (and differences from) other familiar products, processes, and settings. This information becomes part of the database for future comparisons. Factors and parameters to consider include: - application factors - customer factors - people factors - methods - resource factors - tools - process factors - constraints Measurement Goal: Template Template for measurement (sub)goal: Object of interest: Purpose: the in order to it. Perspective: Examine the from the point of view of (the). Environment:,,,,,,,, far@ucalgary.ca 89 far@ucalgary.ca 90

16 Object of Interest: Development Team Purpose: Determine a way to improve the productivity of our development team by evaluating their current productivity. Perspective: Examine the team personality factors, expertise of the development organization, team s analysis and design techniques, knowledge of programming languages of our development team from project manager s point of view. Environment & Constraints: Payroll applications programming in C software developers with 5 or more years experience in C++ Customers are businesses Do not maintain a reusable module database. Examine new projects completed and sold from 1/1/1995 to 31/12/2002. Object of Interest: Working Environment Purpose: Evaluate the working environment in order to identify opportunities in improving the productivity of our development team. Perspective: Examine the ratio of work time to break time of our employees, the accommodations, incentives, and extra-curricular activities offered to our employees, and the workspace (room and desk size, ventilation) where our employees work from the point of view of the employees themselves. Environment & Constraints: Payroll applications programming in C software developers with 5 or more years experience in C++ Customers are businesses Do not maintain a reusable module database. Examine new projects completed and sold from 1/1/1995 to 31/12/2002. far@ucalgary.ca 91 far@ucalgary.ca 92 Object of Interest: CASE Tools Purpose: Evaluate the impact of various CASE tools on the productivity of the development team. Perspective: Examine the effectiveness of using various CASE tools to help in the development of our product from the point of view of the developers and testers. Environment & Constraints: Payroll applications programming in C software developers with 5 or more years experience in C++ Customers are businesses Do not maintain a reusable module database. Examine new projects completed and sold from 1/1/1995 to 31/12/2002. far@ucalgary.ca 93 GQ(I)M: Review Business Goal GQIM Process Measurement Plan 1. Identify business goals (a list of goals in NL) 2. Identify what you want to know or learn in order to achieve the goals (what is? who is? in what way is involved?) (entity - question checklist) 3. Identify subgoals (subgoal - question list) 4. Identify entities and attributes related to subgoals (entities - attributes) 5. Formalize measurement goals (MGs) (MG: entity, purpose, perspective and environment) Already presented in the previous session! far@ucalgary.ca 94 GQ(I)M: Next Steps /1 6. Identify quantifiable questions and indicators that will be used to achieve measurement goals. 7. Identify data elements that will be collected to construct the indicators. 8. Define measures to be used, and make these definitions operational. 9. Identify actions that will be taken to implement the measures. 10. Prepare a plan for implementing the measures. GQ(I)M: Next Steps /2 The GQ(I)M method begins with identifying business goals and breaking them down into manageable subgoals. It ends with a plan for implementing well-defined measures and indicators that support the goals. It can also maintain traceability back to the business goals far@ucalgary.ca 95 far@ucalgary.ca 96

17 Step 6: Identify Indicators /1 Step 6: Identify Indicators /2 What is an indicator? Indicator is a display of one or more measurement results that is designed to communicate or explain the significance of those results to the user. Why is it useful? Seeing how measurement data will be displayed helps clarify exactly what must be measured How to proceed? 1. Select a measurement goal 2. Identify quantifiable questions related to this goal that you would like to be answered. 3. Prepare sketches for displays (indicators) that will help you address your questions. 4. Prioritize the indicators and identify the ones that will be most useful to you. far@ucalgary.ca 97 far@ucalgary.ca 98 Quantifiable Question What is quantifiable question? How is it different from questions in Step 2? Question: Is there sufficient code inspection? (addresses a generic class of entities; answer: not necessarily numeric or logic) Quantifiable Question: What percentage of the code in Module A is inspected? Measurement Goal 1: Development Team Questions: questions that we would like answered are: What are the personality factors of all the development team members? What is the rate at which individual engineers involved in software development produce software and associated documentation? What design and analysis techniques are most effective in product development? Is the development team structured to effectively utilize members skills? How many years of experience do the team members have and in what domain? (addresses a specific entity; answer: numeric or logic) far@ucalgary.ca 99 far@ucalgary.ca 100 Measurement Goal #9: CASE Tools Questions: How much would CASE tools cost? What percentage of the budget is available for tools? Is the increase in productivity offset by the cost of the tools? How long will we have to train employees with the tools? G9 / Ind. b Measurement Goal #9: How much would CASE tools cost? Cost 1000 $CDN Case Tool #1 Case Tool #2 Case Tool #3 far@ucalgary.ca 101 far@ucalgary.ca 102

18 : G9 / Ind. a Measurement Goal #9: What percentage of the budget is available for tools? Maintenance Case Tool Requirements G9 / Ind. c & d Measurement Goal #9: Is the increase in productivity offset by the cost of the tools? Productivity (LOC) before after Time Test Design Average Cost per KLOC Case tool cost LOC cost Coding far@ucalgary.ca 103 without with far@ucalgary.ca 104 : G9 / Ind. e Measurement Goal #9: How long will we have to train employees with the tools? Step 7: Identify Data Elements At this Step we must identify: Data elements that must be collected to construct the indicators identified in Step 6 Definition of data elements, so that the indicators will show what they are supposed to show. Days of training Case Tool #1 Case Tool #2 Case Tool #3 far@ucalgary.ca far@ucalgary.ca 106 Identify Data Elements (cont d) Data elements identification involves: Preparing a list of data items (attributes). Preparing a checklist cross-referencing data items and indicators, i.e., which data element is used by which indicator. Definitions for data items including scales, ranges and precision will be added in the next step. List the data elements vs. the indicators identified in Step 6. Required Data Elements Indicators far@ucalgary.ca 107 far@ucalgary.ca 108

19 Step 8: Define Measure /1 Step 8: Define Measure /2 What is a measure definition? A measure definition is a semi-formal specification for the object to be measured. Why is it useful? It is extremely useful to clarify the implicit assumptions, what is included and what is not in the measurement After the measures are identified they must be defined clearly Definitions must indicate: Name and short description Scale Range of variation Precision required Implicit and explicit assumptions related to measurement, what is included and what is not. Anything else that may help interpret the measurement procedure and values correctly far@ucalgary.ca 109 far@ucalgary.ca 110 Step 8: Example /1 Measuring the height of school children, between the ages of 6 and 12 Measure Definition 1: Height is standing height, measured in inches. Measure Definition 2: Height is standing height (exclusive of piled up hair and hats), measured in inches, bare foot, on a calibrated scale, by a trained nurse, between 8 and 9 o'clock in the morning. Which set of height measurements is expected to be more reliable, more consistent across schools, and more fit for interpretation? Step 8: Example /2 The same applies to definitions such as: The measure for software size is the number of non-commented, nonblank, executable source statements. The measure for cost is the total direct labour hours from the start of the project. These are ill-specified examples of definitions of measures that definitely lead to misinterpretations of measurements and recorded values. far@ucalgary.ca 111 far@ucalgary.ca 112 Step 8: Define Measure /3 Measures can be best defined using Definition checklists & descriptive forms Advantages: Making explicit many issues, assumptions and decisions that usually go unrecorded. far@ucalgary.ca 113 far@ucalgary.ca 114

20 Step 9: Identify Actions to Implement Measures This step is to assemble information about the current status and use of the measures, so as to prepare a plan for implementing the measures defined, through: Analysis & Diagnosis & Actions far@ucalgary.ca 115 far@ucalgary.ca 116 Step 9.1: Analysis Analysis means identifying the measures that the organization is using now and understanding how it is collecting them using questions such as: What data elements are required for our goal-driven measures? Which data elements are collected now? How are they collected? What are the processes that provide the data? How are the data elements stored and reported? Step 9.2: Diagnosis Diagnosis means evaluating the data elements that the organization is collecting now, determining how well they meet the needs of new measurements, and proposing appropriate actions for: Using the data, adapting data to the needs, adapting needs to the data and obtaining what is missing. By asking questions such as: What existing measures and processes can be used to satisfy data requirements? What elements of measurement definitions or practices must be changed or modified? What new or additional processes are needed? far@ucalgary.ca 117 far@ucalgary.ca 118 Step 9.3: Action Action means translating the results of the analysis and diagnosis into implementable steps. Action identifies elements that measurement plan will be built upon. Some pre-planning tasks are: Identify sources of data within existing software process(es). Define methods that will be used to collect and report data. Identify tools that will be required to support collecting, reporting, and storing data. Determine requirements (constraints) for points in time and frequencies of measurement. Document data collection procedures. Identify responsible persons and organizations. Determine where, how, and when to collect and report. Create sketches for the data collection records will be used. Step 9.3: Action Item Checklist Define data elements (Step 8). Define frequencies of collection and points in the process where measurements will be made. Define timelines required for moving measurement results from the points of collection to databases or users. Create forms and procedures for collecting and recording data. Define how data is to be stored and how data will be accessed. Identify who is responsible for designing the database and for entering, retaining, and overseeing the data. Determine who will collect and access data. Assign responsibilities for these actions. Define how data will be analyzed and reported. Identify supporting tools that must be developed or acquired to help you automate and administer the process. Prepare a process guide for collecting data. far@ucalgary.ca 119 far@ucalgary.ca 120

21 Step 10: Prepare a Plan Measurement implementation plan is based on analysis, diagnosis and actions (Step 9) Measurement plan template includes: 1. Objective 2. Description 3. Implementation 4. Sustained operations Step 10: Plan Template / Objective List the principal objectives of this measurement implementation effort. Identify the measures to be implemented, explain why they are important to the organization, and summarize the expected outcomes. far@ucalgary.ca 121 far@ucalgary.ca 122 Step 10: Plan Template / Description 1. Background A brief history of the events that have led to or motivated this plan. Describe the origins of the plan, the work that has been done to date and who participated. Relate the planned actions to other existing or concurrent measurement activities within the organization and (if appropriate) in those of customers or suppliers. 2. Goals List and explain the goals that motivate and guide the activities under this plan by identifying three kinds of goals: (a) business goals, (b) measurement goals, and (c) the goals of this plan. Step 10: Plan Template / Description 3. Scope Relate the measures that this plan implements to the measurement goals. Do the measures apply to new projects only? To development projects? To large or small programs? To only certain divisions or departments? Who are the major stakeholders? Who will be affected by the measurement practices, processes, and methods? Who will use the results? Identify the time span over which this plan is to be effective. far@ucalgary.ca 123 far@ucalgary.ca 124 Step 10: Plan Template / Description 4. Relationship to other software process improvement efforts Describe how the measurement efforts in this plan relate to other process improvement activities at the organization, such as the CMM, the Baldrige Award, or ISO 9000 certification. 5. Relationship to other functional activities Describe how the measurement efforts in this plan relate to other functional groups and activities at the organization, such as cost estimating, time and effort reporting, cost accounting, and quality assurance. Step 10: Plan Template / Implementation Describe the actions that are to be taken to implement the measures identified in the Description. It includes detailed information on Activities, products, and tasks Schedules Resources Responsibilities Measurement and monitoring Assumptions Risk management far@ucalgary.ca 125 far@ucalgary.ca 126

22 Step 10: Plan Template /6 3.1 Activities, Products, and Tasks Describe how the effort is to be accomplished. Partition the effort into manageable activities, products, and tasks that can be used as a basis for planning, reporting, and control. For each activity, product, or task, state the objective and identify the principal subtasks. Where possible, identify entry and exit conditions that will determine start and completion of the task. 3.2 Schedule Describe when each activity, product, or task is to be accomplished. Use PERT charts, or alternative displays to describe sequences and dependencies. Translate key actions, events, and deliverables into milestones so that performance can be tracked against plans. Step 10: Plan Template /6 3.3 Resources Describe resources that are being allocated to this effort, such as: personnel, money, facilities, computer resources, etc. 3.4 Responsibilities Name individuals or groups that will be responsible for overseeing, planning, implementing, managing, approving, and funding this effort. Assign responsibility and authority for acquiring tools, for training, and for implementing and operating databases. 3.5 Measurement and Monitoring Describe how the progress of implementing these measures will be measured, analyzed, and reported. Identify replanning points and describe how significant schedule deviations or changes and revised funding needs will be handled. far@ucalgary.ca 127 far@ucalgary.ca 128 Step 10: Plan Template /7 3.6 Assumptions Identify key assumptions upon which this plan is based. Key assumptions are ones which, if not satisfied, pose risks for successful implementation. 3.7 Risk management Describe how you will identify, assess, track, and do contingency planning for the risk factors associated with the measurement implementation efforts covered by this plan. Describe actions that will be taken to monitor the assumptions, and provide mechanisms for reacting if assumptions are not met. Identify all places where planned schedules and resources differ from estimates and describe actions being taken to make the planned outcomes achievable. Step 10: Plan Template / Sustained operations Describe the actions that will be taken to sustain and use the measures implemented. Assign resources and responsibilities and make provisions for continuing evolution. Describe the practices that will be used to evaluate and monitor the effectiveness of the measures and to assess their business value and their effects on organizational performance. Alternatively, if appropriate, provide direction and resources for preparing an operational plan for sustaining the collection, use, retention, evolution, and evaluation of these measures. far@ucalgary.ca 129 far@ucalgary.ca 130 Exercise /1 Exercises What different attributes might be measured by the followings: The number of faults in program P using a set of test cases created specifically for the program P. Failure (rate) intensity The number of faults in program P using a standard set (benchmark) of test data. Code efficiency The number of faults in program P by programmer A during 1 hour. (Individual) test efficiency far@ucalgary.ca 132