Ulrike Dowie, Lars Karg SAP AG Software & Systems Quality Conferences 25. April 2007

Transcription

1 A Systematic Approach for Test Effort Estimation Model Selection Ulrike Dowie, Lars Karg SAP AG Software & Systems Quality Conferences 25. April 2007

2 Motivation and Aims of Our Approach Criteria to Examine Existing Models Selection Approach A Partly Fictitious Case Study Evaluation

3 Motivation and Aims Deadlines and budgets are missed Guesstimation is apparently inadequate to plan test Models and methods for test effort estimation exist, but Which one to choose? Aims: Develop a systematic selection approach Tailored to domain, organization, and project To facilitate comparison between existing models or methods To reduce selection effort after first use of the approach To select systematically and objectively SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 3

4 Conceptual Framework Support Assessment and Selection Test Goals Test Parameters Influence Result In Support and Help to Reach Test Techniques Efficiency and Effects Test Restrictions Affect SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 4

5 Motivation and Aims of Our Approach Criteria to Examine Existing Models Selection Approach A Partly Fictitious Case Study Evaluation

6 Criteria to Examine and Compare Existing Models (1/2) 1. Requirements on the model Falsifiability Assumptions, hypotheses can be refuted by experience portability Objectivity Model is based on formal process, different persons arrive at the same results Model maturity Number of practical applications, diversity of application: e.g., number of different organizations, different domains Usage experience User satisfaction model use continued Usability Comprehensibility, adaptability, applicability Project control Feedback loop, alternative actions suggested? Programming language independence SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 6

7 Criteria to Examine and Compare Existing Models (2/2) 2. Match models and organizational context/project Goals Is a goal-oriented process modeled? Can model user choose among different goals? Restrictions and parameters Process model, programming language, available tool support Historical data quality and quantity (metrics, number of projects, etc.) Human resources (statistical kwledge, experience, etc.) Test characteristics (comparability of test cases, structural versus functional testing, etc.) Assumptions concerning effects Are assumptions valid in the organizational/project context? Results traceable back to causes SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 7

8 Selection Approach Preparation Test Effort Estimation Models (TEEMs) Domain Restrictions TEEMs Applicable to Domain Org. Goals & Restrictions TEEMs Applicable to Organization Project Goals, Restrictions & Parameters TEEMs Applicable to Project with Ranking SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 8

9 Motivation and Aims of Our Approach Criteria to Examine Existing Models Selection Approach A Partly Fictitious Case Study Evaluation

10 Case Study (1/4) Preparation Search for Test Effort Estimation Models (TEEMs) Analysis of TEEMs according to criteria (see next slide) Determination of critical criteria Falsifiable (assumptions can be checked against real conditions) Usage experience: must be positive (model is still being used) Model maturity: practically applied Understandable: estimation results must be traceable Adaptability: parameter determination must be clear Language independent Feedback loop to control test efforts: nice to have SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 10

11 Case Study (2/4) Model Is falsifiable Model Is objective Usage Experience Model Maturity Usability Understandable Adaptability Tool Support Feedback Loop PC Alternative Measures Given Language Independent Calzolari (1998) low 3 Cangussu (2002) low 4 pa. Models Nageswaran (2001) Pensyl (2002) partly medium low 3 2 pa. pa. Singpurwalla (1991) partly ne 1 Sneed (2006) medium 2 SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 11

12 Case Study (3/4) Domain restrictions Competitive situation: supplier oligopoly budget and schedule constraints must be kept Bug corrections possible after market release Highly variable product usage t all process chains can be tested prioritization of test cases is important Organizational goals (especially regarding TEEMs) Model results as additional input for resource planning model must be applicable early in the product life cycle Organizational restrictions (especially regarding TEEMs) Development process: variant of the V-model Languages: object-oriented Test cases: project-specific extent/coverage t comparable Use case/function point counts: t available SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 12

13 Case Study (4/4) Fulfillment of critical criteria No model is entirely falsifiable use partly falsifiable models One model with positive usage experience select for further analysis Matching models with domain restrictions Competitive situation budget and schedule constraints are considered by the remaining model Matching models with organizational goals Model can be used early in the life cycle (requirements must be defined) Matching models with organizational restrictions V-model is supported, object-oriented languages are supported Test cases should require similar effort assumption violated No TEEM fits domain and organizational restrictions SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 13

14 Motivation and Aims of Our Approach Criteria to Examine Existing Models Selection Approach A Partly Fictitious Case Study Evaluation

15 Evaluation: Benefits and Use of the Approach Organizational and project goals need to be identified Missing or unclear goals become obvious Valuable input for project team s work Time dedicated to test effort estimation Reasonable, detailed effort estimation is facilitated Effort estimation t a single person s task but a group task Objectivity instead of subjectivity Activities won t be forgotten as easily (due to cross-checks) Project, product, and team characteristics required as input Project is analyzed more thoroughly and carefully More reliable test effort estimation (even when model is appropriate) SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 15

16 Evaluation: Lessons Learned Identify successfully applied models only (practical application by itself is insufficient) Use elimination criteria instead of preselection at domain level Contact peers (other software developing organizations) and share experience regarding test effort estimation models Evaluate local influencing factors of the test effort Collect data Interview long-time experienced test coordinators/managers Analyze project data and interviews (graphically, statistically) SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 16

17 Evaluation: Further Research Directions Framework needs to be applied in several projects To reduce time required To find objective measurements to replace subjective measurements To extend or reduce requirements More models need to be identified and analyzed Falsifiable models: t pretending general applicability, allowing to determine all parameter values locally Successfully applied models Analytical models explaining test effort and its influencing factors are needed To be cross-checked in the organization SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 17

18 Bibliography F. Calzolari, P. Tonella, G. Antoniol, G., Dynamic Model for Maintenance and Testing Effort, Proceedings of the International Conference on Software Maintenance (ICSM'98), pp , Joao W. Cangussu, Raymond A. DeCarlo, Aditya P. Mathur, A formal model of the software test process, IEEE Transactions on Software Engineering, Vol. 28, No. 8, pp , Aug Suresh Nageswaran, Test effort estimation using use case points, Presentation at the Quality Week 2001, San Francisco, Jim Pensyl, Test effort estimation for the design and development of manual functional/regression test scripts, 2002, URL: Nozer D. Singpurwalla, Determining an optimal time interval for testing and debugging software, Vol. 17, No. 4, pp , Apr Harry M. Sneed, Stefan Jungmayr, Produkt- und Prozessmetriken für den Softwaretest, Informatik Spektrum, Vol. 29, No. 1, pp , Feb SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 18

19 Copyright 2007 SAP AG. All Rights Reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior tice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iseries, pseries, xseries, zseries, System i, System i5, System p, System p5, System x, System z, System z9, z/os, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix, i5/os, POWER, POWER5, POWER5+, OpenPower and PowerPC are trademarks or registered trademarks of IBM Corporation. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Techlogy. Java is a registered trademark of Sun Microsystems, Inc. JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for techlogy invented and implemented by Netscape. MaxDB is a trademark of MySQL AB, Sweden. SAP, R/3, mysap, mysap.com, xapps, xapp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG. This document is a preliminary version and t subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is t intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please te that this document is subject to change and may be changed by SAP at any time without tice. SAP assumes responsibility for errors or omissions in this document. SAP does t warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but t limited to the implied warranties of merchantability, fitness for a particular purpose, or n-infringement. SAP shall have liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall t apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is t affected. SAP has control over the information that you may access through the use of hot links contained in these materials and does t endorse your use of third-party Web pages r provide any warranty whatsoever relating to third-party Web pages. SAP AG 2007, Test Effort Estimation Model Selection / Ulrike Dowie, Lars Karg / 19