Proposal for Master Thesis in Software Engineering

Transcription

1 Proposal for Master Thesis in Software Engineering Base information Student 1 Name, and P.Nr.: A.K.M. Moinul Islam, moib08@student.bth.se, P154 Student 2 Name, and P.Nr.: Michael Unterkalmsteiner, miun08@student.bth.se, P112 Title (preliminary): The Empirical Evaluation and Refinement of A Software Process Improvement Measurement Framework (SPI-MeF) Link to project home page (if any): N/A Academic Advisor: Dr. Tony Gorschek Faculty Reviewer: Anonymous Start- and end-date: 19 th January th June 2009 (Spring semester 2009) Thesis type (research / industrial, theoretical / empirical): Industrial, Empirical Student 1 suitability Software engineering course credits completed at BTH (total): _30_ ECTS credit points Software engineering courses completed at BTH relevant for thesis work: Advanced Software Engineering Topic Software engineering course credits completed at UKL (total): 62_ ECTS credit points Software engineering courses completed at UKL relevant for thesis work: Project Management and Quality Assurance Quality Management of Software and Systems Process Modeling Empirical Model Formation and Methods Student 2 suitability Software engineering course credits completed at BTH (total): 30 ECTS credit points Software engineering courses completed at BTH relevant for thesis work: Advanced Software Engineering Topic Software engineering course credits completed at FUB (total): 47_ ECTS credit points Software engineering courses completed at FUB relevant for thesis work: Software Quality Management Page 1

2 Definitions Terms SPI Initiative Evaluate Context Evaluation method Metric Effective evaluation Conceptual framework Applicability Definitions All software engineering methods or activities which are intended to improve the performance of the software process. SPI initiatives can be categorized into frameworks (e.g. CMM, CMMI, SPICE, QIP, Six Sigma, etc), software engineering practices (e.g. inspections, test-driven development, etc.) or tools that support software engineering practices. To measure or assess the value of interest. The environment in operation which varies in several aspects. In SPI context, it refers to size, domain, SPI initiative, etc. of the organization. A systematic determination of values of interest using a set of rules. In SPI context, it refers to the method used to measure the outcome of SPI initiative. Examples are comparison using pre and post data, survey, statistical analysis, etc. A real objective measurement describing the structure or content of software products or software processes that has a standard unit of measure. An evaluation that is performed in an accurate way to assess the intended outcome of interest. A reusable design of framework which can be instantiated using a certain set of parameters in order to get a specific framework that can be used in practice. Flexible in terms of adapting to different industrial contexts and reusable in terms of reusing without significant changes in similar industrial contexts. Background With the increasing importance of software products in industry as well as in our everyday's life, the process of developing software has gained major attention by software engineering researchers and practitioners in the last three decades. Software Process Improvement (SPI) is the research area which addresses the assessment and improvement of the processes and practices involved in software development. The basic principle behind process improvement is the iterative execution of the following four steps: Assess the current status of a process Elaborate an improvement plan Implement the plan Evaluate the improvement The two dominant streams in software process improvement differ conceptually in the way how the to-be-improved processes are identified. SPI frameworks based on the bottom-up approach assume that an in-depth understanding of the processes, products and company goals is needed in order to implement an effective improvement program [1]. This is in Page 2

3 contrast to the top-down approach where the SPI framework specifies some generally accepted standard processes against which the organizations' processes are benchmarked. The elimination of the differences between the standard process and the actual one is then the process improvement [1]. Investment on improving the software process has increased significantly, and several research papers document SPI s effectiveness. The SPI literature also contains many case studies of successful companies and descriptions of their improvement programs. One important aspect in conducting a software process improvement initiative is the measurement of its effects on the process itself and the produced artifacts. The measurement of the software process is a substantial component in the endeavor to reach predictable performance and high capability and to ensure that process artifacts meet their specified quality requirements [2, 3]. Software measurement is acknowledged as essential in the improvement of software processes and products since if the process (or the result) is not measured the SPI effort could address the wrong issue. SPI practitioners have found it difficult to develop and implement effective performance measurement programs for SPI, in part because guidelines for conducting SPI measurements are scarce [4]. Mendonsa et.al. defines the required properties of a measurement framework as a set of related metrics, data collection mechanisms, and data uses inside an organization [5]. Therefore, a measurement framework that can address the aforementioned problems could guide SPI practitioners in introducing an effective evaluation program in their organization. Such a framework could provide concrete evidence of the SPI initiatives' outcome, visible to the relevant and interested stakeholders. An example of such a measurement framework is Software Process Improvement Measurement Framework (SPI-MeF). It is a conceptual framework that is designed to effectively implement a measurement program for SPI initiatives. As with any other software engineering technique, method or tool, this framework requires an empirical evaluation in order to assess its usefulness and applicability in the industry. According to Kitchenham there are nine evaluation types for evaluating software engineering methods and tools [6]. Selecting which evaluation method to use is an important aspect for obtaining a valid evaluation result. Kitchenham explained criteria for selecting evaluation method [7, 8]. For the evaluation of SPI-MeF we will use qualitative effect analysis, which is a subjective assessment by experts who report the quantitative effects methods or tools [6]. Aims and Objectives The aim of this thesis project is to empirically evaluate SPI-MeF and incorporate improvements to refine the measurement framework. This will be achieved by addressing the following objectives: Gain an understanding of SPI-MeF which supports the effective evaluation of SPI initiatives outcome. Design of the evaluation method for SPI-MeF. Determine the degree of applicability of SPI-MeF to the industry. Gather recommendations for improvements to refine SPI-MeF. Incorporate the recommended refinements into SPI-MeF. Page 3

4 Research Questions RQ1: To which extent can SPI-MeF improve the evaluation effectiveness of SPI initiatives' outcome, based on expert opinion collected during industry evaluation? RQ2: To which extent is SPI-MeF applicable in practice, in terms of its flexibility and reusability in different industrial contexts, based on expert opinion collected during industry evaluation? RQ3: What are the major insufficiencies of SPI-MeF in relation to industry use, based on expert opinion collected during industry evaluation? RQ4: What are the improvements to SPI-MeF that are recommended by industry experts after the evaluation? RQ5: How can refinements be incorporated into SPI-MeF based on the recommended improvements in RQ4? Expected Outcomes The expected outcomes of this thesis are: Evaluation of SPI-MeF in terms of its applicability to the industry. Feedbacks from industry and experts opinion on SPI-MeF. Refined SPI-MeF incorporating the recommendations from the industry evaluation. Research Methodology Qualitative Effect Analysis Kitchenham et. al. proposed two main types of evaluations in their evaluation framework for software engineering methods (DESMET). The quantitative approaches aim at establishing measurable effects of using a method or tool; qualitative ones aim at establishing method or tool appropriateness, e.g. how well they fit the needs and culture of an organization [9]. Some methods combine both quantitative and qualitative evaluation (hybrid methods). In total, DESMET contains nine distinct evaluation methods: i. Quantitative experiment ii. Quantitative case study iii. Quantitative survey iv. Quantitative screening v. Qualitative experiment vi. Qualitative case study vii. Qualitative survey viii. Hybrid 1: Qualitative effects analysis ix. Hybrid 2: Benchmarking Page 4

5 Qualitative effects analysis, the evaluation method chosen for this thesis projects, is applicable under the following conditions [10]: Availability of expert opinion assessments of methods/tools We assume that we can acquire expert opinion from within the company where the framework is evaluated. In case this is not possible due to the lack of internal experts, we can consider acquiring the opinion of an external consultant. Lack of stable development process This property allows us to select evaluation candidates from a larger pool, since the evaluation method doesn't require a stable development process. Requirements to mix and match methods/tools This property applies to our context in the sense that the mixing of different evaluation methods and/or metrics may be required in order to achieve (match) a certain goal, e.g. showing the improvement in productivity. Besides the above shown fitness of the qualitative effects analysis in the context of this research, the evaluation can generally be conducted in a relative short time if a database of expert opinion about method or tool impacts is already available [10]. Since this is not the case in our study, we will have to plan-in considerable more time for the evaluation than estimated by Kitchenham (a few days) [10]. Qualitative effects analysis has, relative to the other evaluation methods, a very high risk of drawing incorrect conclusions from the evaluation [10]. This is due to the subjective view of a single expert (in our case a single expert within an organization). We will compensate this risk by taking into account expert opinion from an external consultant when consolidating the recommendations. Semi-structured Interview The feedback from the industry experts will be acquired by semi-structured interviews. Such interviews contain specific questions (to gather the anticipated information) and open-ended questions (to gather unforeseen information) [11]. The qualitative data will be used to assess the appropriateness and applicability of the measurement framework. Quantitative data will be collected if the evaluated organization has already a measurement program in place and its evaluation methods/metrics can be compared with the ones proposed by the measurement framework (e.g. number of metrics in place vs. proposed metrics, effort to collect data for metrics in place vs. estimated effort collect data for proposed metrics). Conceptual Analysis In conceptual analysis, a concept is chosen for investigation, and the analysis involves quantifying and tallying its presence [12]. According to M. Beaney, "conceptual analysis consists primarily in breaking down or analyzing concepts into their constituent parts in order to gain knowledge or a better understanding of a particular philosophical issue in which the concept is involved" [13]. Conceptual analysis will be used as one of the research method in the evaluation of the SPI- Page 5

6 MeF. After gathering information from interview of industry experts, the gathered information will be conceptualized combining the existing knowledge of the framework and then be broken down into constituent parts to gain better understanding of the concept. This will help to summarize the effectiveness, applicability and insufficiency of SPI-MeF and then to perform feasible refinements. However, findings or the outcomes of the conceptual analysis need further empirical evaluation as conceptual analysis does not give empirical evidence and may not be based on realistic evaluation. Evaluation of SPI-MeF will be performed using the following steps: 1. Evaluating the effectiveness, applicability and insufficiencies of SPI-MeF: In the first step of evaluation, requirements will be gathered from the organization where the measurement framework will be evaluated. Thereafter, a specific framework will be instantiated based on the input from the organization. In case of an existing measurement program in the organization, its metrics and evaluation methods will be compared with the ones proposed by SPI-MeF. Then a semi-structured interview is conducted and qualitative effect analysis is done on the gathered data to evaluate the effectiveness, applicability and Page 6

7 insufficiencies of SPI-MeF and to summarize the recommended improvements. 2. Conceptual analysis of the recommended improvements: In this step, the areas of improvement in the framework will be identified through conceptual analysis based on the consolidated recommendation of the evaluation findings of previous step. 3. Refinement of SPI-MeF: The outcome of conceptual analysis will be used to define the areas of improvement and properties of the framework. Thereafter, refinement of the framework will be conducted by incorporating the feasible recommendations. Research Questions Aims Methodology RQ1: To which extent can SPI- MeF improve the evaluation effectiveness of SPI initiatives' outcome, based on expert opinion collected during industry evaluation? Investigating how well a particular instance of SPI measurement framework can improve the effectiveness of measuring the outcome of SPI initiatives. Semi-structured interview + Qualitative effect analysis RQ2: To which extent is SPI- MeF applicable in practice, in terms of its flexibility and reusability in different industrial contexts, based on expert opinion collected during industry evaluation? RQ3: What are the major insufficiencies of SPI-MeF in relation to industry use, based on expert opinion collected during industry evaluation? RQ4: What are the improvements to SPI-MeF that are recommended by industry experts after the evaluation? RQ5: How can refinements be incorporated into SPI-MeF based on the recommended improvements in RQ4? To investigate the applicability of a particular instance of the measurement framework in a certain industrial context. To find out the difficulties faced when applying the measurement framework in practice. Gathering and consolidating the feedback from different experts to idenitfy the areas of improvement in the measurement framework. To find out the ways to incorporate the feasible improvement recommendation got by the analysis of the identified areas improvement in the SPI-MeF and to actually implement them. Semi-structured interview + Qualitative effect analysis Semi-structured interview + Qualitative effect analysis Qualitative effect analysis + Conceptual analysis Conceptual analysis Page 7

8 Risks Risks Identifying the appropriate candidates for evaluators Possible Solutions i) Choosing the organizations that vary in types of SPI initiatives and contexts ii) Choosing the right candidates for experts opinion who have extensive experience within the area of interest. Assess the research contributions of the candidate, e.g. by looking at the number of publications in the related fields. Unable to conduct face-to-face meeting with evaluators Unable to identify enough contacts for industry evaluations If the evaluators don't have time for formal interviews, the alternatives are to conduct offline interviews by s or phone calls. Alternative is to find historical dataset of organizations that are made public. Time Plan Scheduled Milestones for Thesis Work: : Start writing the thesis proposal : First draft of thesis proposal : Final draft of thesis proposal : Study on SPI-MeF : Start planning for industry and experts evaluation : Organization candidates and experts identified : Industry evaluation and interview guide finished : Start to conduct evaluations and interviews : All evaluation and interviews conducted and transcribed : Start analysis and consolidating of recommended improvements : Start identifying improvement areas to SPI measurement framework : End of SPI measurement framework refinement : First draft of thesis report to supervisor : Supervisor tells examiner we are ok for presentation : Updated final draft sent to opponents : Thesis Presentation : Final thesis updated, approved and sent to examiner Page 8

9 References [1] Thomas M., McGarry F., "Top-down vs. bottom-up process improvement", IEEE Software, v 11, n 4, July 1994, pp [2] Paulish, D.J., Carleton, A.D., "Case studies of Software-process-improvement Measurement", Computer, v 27, n 9, Sept, 1994, pp [3] W. A. Florac, A. D. Carleton, "Measuring the Software Process: Statistical Process Control for Software Process Improvement" (SEI Series in Software Engineering), Addison-Wesley Professional, 1999 [4] Iversen J. and Ngwenyama O., "Problems in measuring effectiveness in software process improvement: A longitudinal study of organizational change at Danske Data", International Journal of Information Management, Volume 26, pp , 2006 [5] M. Mendonsa, V. R. Basili, I. S. Bhandari, and J. Dawson, "An approach to improving existing measurement frameworks", IBM Systems Journal, Volume 37, Issue 4, pp , 1998 [6] B. Kitchenham, Evaluating software engineering methods and tool. 1. The evaluation context and evaluation methods, SIGSOFT Software Engineering Notes, vol. 21, Jan. 1996, pp [7] B.A. Kitchenham, Evaluating Software Engineering Methods and Tool - Part 2: Selecting an Appropriate Evaluation Method-Technical Criteria, Software Engineering Notes, vol. 21, 1996, p. 11. [8] B. Kitchenham, Evaluating software engineering methods and tool.3. Selecting an appropriate evaluation method-practical issues, Software Engineering Notes, vol. 21, Jul. 1996, pp [9] Mendoza, L.E., Perez, M.A., Rojas, T., Griman, A., De Luca, L.A., "Selecting tools for software quality management", (2002) Software Quality Professional, 4 (4), pp [10] Kitchenham B., "DESMET: A method of evaluating Software Engineering methods and tools", Technical Report TR96-09, 1996 [11] Hove, S.E., Anda B., "Experiences from Conducting Semi-Structured Interviews in Empirical Software Engineering Research", Proceedings - 11th International Software Metrics Symposium, p , [12] Palmquist, M. (2006). Overview: Content Analysis. Colorado State University Conducting, Content Analysis Writing Guide [Online]. Available: [Accessed: Feb 15, 2009] [13] M. Beaney, "Analysis", The Stanford Encyclopedia of Philosophy, Published April 7, 2003 [Online]. Available: The Stanford Encyclopedia of Philosophy, [Accessed: Feb 15, 2009] Page 9