Proposal for Master Thesis in Software Engineering

Transcription

1 Proposal for Master Thesis in Software Engineering Base information Student 1 Name, and P.Nr.: Jan Schulte, jasd08@student.bth.se, p834 Student 2 Name, and P.Nr.: Philip Preissing, philippreissing@googl .com, external (TU München, Germany) Title (preliminary): A Software V&V Management Framework for the Space Industry Link to project home page (if any): Academic Advisor: Dr. Robert Feldt (BTH), Dr. Sira Vegas (UPM) Industry Advisor: Annalena Johansson (RUAG Aerospace Sweden AB) Faculty Reviewer: Anonymous Start- and end-date: 2009/02/01 to 2009/08/01 Thesis type (research / industrial, theoretical / empirical): industrial, theoretical and empirical Student 1 suitability Software engineering course credits completed at BTH (total): 30 ECTS credit points Software engineering courses completed relevant for thesis work: Modelos y Metodos para la Evaluacion y Mejora de Procesos Software process improvement) Caso Practico de Calidad de Software (Software quality) Metricas de Software (Software metrics) Projekt Management Student 2 suitability Software engineering courses completed relevant for thesis work: Projektmanagement und Qualitätssicherung (Project management and quality assurrance) Projekt Management Advanced seminar course software quality Background Software for space applications has special requirements in terms of reliability and dependability. Therefore, existing software engineering methods are not sufficient. The European Cooperation for Space Standardization (ECSS) has developed a set of standards to cope with these issues. As any other industry, the space industry is faced with a pressure to deliver higher quality software fast and with less effort and thus cost. In a research project together with RUAG Aerospace Sweden AB and Swedish Space Corporation (SSC), BTH is trying to optimize Page 1

2 Verification and Validation activities (VAs) wrt. effectiveness and efficiency. The companies feel that they find the same defects with multiple VAs and therefore see a chance to reduce this redundancy without negatively impacting quality. In the first part of the collaboration with RUAG and SSC, BTH has focused on analyzing the current situation of both companies. They examined the companies way of work and the challenges they face regarding verification and validation [8] [9]. A master thesis [10], conducted within this project, found three main problems: faults slip through between development phases, inappropriate selection of VAs, and ECSS standards. Instead of focussing on a single verification and validation activity, research [1] [2] and industry [3] shows that combining different VAs can be more efficient in finding defects. To overcome the constant market pressure thats omnipresent in industry, further investigation is necessary to guide industry on how to select VAs to maximise the defect detection efficiency while minimising the effort spent. Some initial work that has been done towards the creation of a comparison framework is presented in the following. Wojcicki and Strooper present in [1] an iterative selection strategy (ISS) for verification- & validation- activities. The VAs are hereby firstly selected by their efficiency in order to maximise completeness, and secondly by the effort, i.e. the cost, they require. Based on the data collected while applying the technique, the selection is refined iteratively. In [5] the concept of Fault-Slip-Through (FST) is presented. In this approach, the faults found are categorized according to which phase they belong to, i.e. in which VA they should have been found. Based on these findings and the effort of the VA, the improvement potential for each activity is calculated. Wagner [3] proposes a more analytical approach. The defect detection techniques are hereby compared using an economical metric, namely the return on investment (ROI). The model also considers the effect of combining different defect detection techniques. A framework for the comparison of testing activities and formal verification is presented in [7]. However, this approach focuses more on studying the synergy and relationship of these two activities and doesn t give any advice on how this combination can be optimized. There exists an IEEE Standard 1061 [11] related to software metrics and measurement. It defines a methodology for establishing quality requirements and finding, measuring and validating appropriate software product and process metrics. This can help in judging the effort and defect detection capability of VAs. This thesis targets the first two of the previously mentioned problems by combining measurements and a selection strategy for VAs in a coherent framework. Today, no such framework exists. It is neither known which measurements and VAs can be used for such a framework, nor how iterations can be used to gradually improve real-world software development processes. Because the framework is going to be used in an industry setting, it has to be as lightweight as possible and should only require a minimal set of changes to the existing processes. Aims and objectives The goal with this project is to create a coherent framework for the management and optimization of software verification and validation activities. Page 2

3 Adapt FST and ISS to the development processes at RUAG Aerospace Sweden AB Creation of a framework that enables the comparison of the effectiveness of different VAs in eliminating different types of software defects and the comparison of the efficiency of different Vas. Validation of the framework at RUAG Aerospace Sweden AB Description of a minimal set of changes that RUAG needs to make to adapt to the framework A consideration of how the framework could be applied to the more agile development processes used at Swedish Space Corporation (SSC). Research questions 1. How can Fault-Slip-Through(FST) and Iterative Selection Strategy(ISS) be combined into a coherent framework? 2. Which alternatives exist to the aforementioned models? 3. Which measurements need to be done to support this framework? 4. How can the framework support parameterization to include existing measures and thus, minimize the measuring effort to be real-world applicable? 5. How can the framework be adapted to a real-world industry setting by only doing minimal changes to the company s processes? 6. How does the framework perform in a real-world industry setting? 7. How can the framework be adapted to a more agile process? Expected outcomes A description of the framework A process guideline for RUAG describing how the framework can be applied to their processes. Validation results gained by interviews and, if possible, by the data measured in a real project A list of recommendations for improving the RUAG development process based on the application of the framework A description on the suitability of applying the framework in a more agile environment Research Methodology First, the expectations and goals of RUAG are collected based on interviews. This provides hints on the important aspects to address in the framework. A literature survey is carried out to reveal the current state-of-the-art including which alternative models exist for the selection of the VAs and the measurement of these. The process documents of RUAG will be analysed to gain an in-depth knowledge of their software development processes. This includes roles, development phases, VAs used in each phase and metrics collected. Furthermore, the existing data and metrics collected for the different VAs will be investigated and evaluated with respect to their suitability for usage in the framework. Since for FST the test strategy has to be known in advance, it is extracted out of RUAG documents and data collected; and then further refined through interviews with RUAG responsibles. Page 3

4 To allow a lightweight application in real industry settings, the framework is parameterized with different measurement levels, e.g. reaching from fully automated to manual data entry. These levels will be investigated under consideration of previous work done at Ericsson. Additionally, the results of the master thesis [8] serve as input for providing preliminary data of the efficiency and effectiveness of the VAs in RUAG. Based on these inputs (the master thesis, the state-of-the-art and alternative models, RUAG processes, the available metrics and data, the expectations and goals of RUAG, the test strategy and the measurement levels), a coherent framework will be developed. It will be documented in a written framework description. It is then determined whether the existing data is sufficient to support the application of the framework. According to the measurement levels, a minimum set of changes to the RUAG processes is developed. Those boil down to a process guideline describing the possible application of the framework at RUAG. The framework will be analysed statically by reviews or if possible through to the application of the framework in a real project and analysing the data obtained after a certain period of time. Based on the feedback gained, the framework will be revised iteratively. Based on the framework and the evaluation a guideline is established giving improvement recommendations to RUAG. Those are evaluated through interviews with RUAG responsibles. In cooperation with SSC it is evaluated how the framework can be adapted to a more agile process using interviews or workshops. A description concerning the suitability is developed including possible necessary adaptations. Risks Threat Possible solution Probability Insufficient data or Data gained from interviews with RUAG in Medium/ insufficient support from the Master thesis [Towards Optimizing High RUAG to measure Verification and Validation Activities in meaningful/significant data Space Industry] will serve as input Dynamic validation of the A static evaluation through reviews will be High framework not possible due to lack of available projects at that time conducted Lack of time Prioritization of the research questions, e.g. by limiting the effort spent on less important tasks (in discussion with supervisors and Low Adaptability of the framework depend on the resources available at RUAG RUAG) Being focussed and effective in the interactions with RUAG Medium/ High Time plan Scheduled Milestones and Meetings: : Start writing the proposal : First draft of proposal to supervisor Page 4

5 : Final draft of proposal to supervisor : End of literature review : Start collecting expectations and goals of RUAG : End collecting : Start of analysis of the software development processes at RUAG : End of analysis : Start data analysis : End data analysis : Start definition of test strategy and measurement classification : End of definition and classification : Start working on the framework : Finish framework and documentation so that it is ready to be used in a project : Start evaluation of the framework : Evaluation finished : Start working on improvement proposals : Preliminary results available for presentation at DASIA : Improvement proposals finished : Start of the evaluation of the framework at SSC : Evaluation of the framework at SSC finished : Evaluation of the recommended improvements : Evaluation finished : Supervisor tells examiner we are ok for presentation : Updated final draft sent to opponents : Thesis Presentation : Final thesis updated, approved and sent to examiner References [1] R. Feldt, E. Ahmad, B. Raza, E. Hult, and T. Nordebäck, "Evolving the ECSS standards and their Use: Experience based on Industrial Case Studies," Data Systems in Aerospace, 2009, in submission. [2] B. Raza, E. Ahmad, R. Feldt, and T. Nordebäck, "ECSS Standard Compliant Agile Development for Dependable Space Software - an Industrial Case Study," 2008, in submission. [3] E. Ahmad and B. Raza, Towards Optimizing Verification and Validation Activities in Space Industry. Ronneby: Blekinge Tekniska Högskola, 2009, Master thesis. [4] B. Littlewood, P. T. Popov, L. Strigini, and N. Shryane, "Modeling the effects of combining diverse software fault detection techniques," IEEE Transactions on Software Engineering, vol. 26, no. 12, pp , [5] B. Kitchenham and S. Linkman, "Validation, verification, and testing: diversity rules," IEEE Software, vol. 15, no. 4, pp , [6] N. Kikuchi and T. Kikuno, "Improving the testing process by program static analysis," in Software Engineering Conference, APSEC Eighth Asia-Pacific, 2001, pp [7] M. A. Wojcicki and P. Strooper, "An Iterative Empirical Strategy for the Systematic Selection of a Combinationof Verification and Validation Technologies," in Fifth Page 5

6 International Workshop on Software Quality, [8] L.-O. Damm, Early and Cost-Effective Software Fault Detection. Ronneby: Blekinge Tekniska Högskola, 2007, Ph.D. thesis. [9] S. Wagner and T. Seifert, "Software quality economics for defect-detection techniques using failure prediction," in 3-WoSQ: Proceedings of the third workshop on Software quality, New York, NY, USA, 2005, pp [10] J. S. Bradbury, J. R. Cordy, and J. Dingel, "An empirical framework for comparing effectiveness of testing and property-based formal analysis," SIGSOFT Softw. Eng. Notes, vol. 31, no. 1, pp. 2-5, [11] IEEE, IEEE Std : Standard for a Software Quality Metrics Methodology Page 6

7 Appendix A Research Methodology Literature survey Data Master thesis RUAG process descriptions Interviews Ericsson Interviews Consideration of alternatives RQ2 Process analysis Data analysis Collect expectations and goals of RUAG Develop test strategy Measurement classification RQ4 Framework development RQ1&3 Framework description Develop miminum set of changes RQ5 Process guideline Interviews Evaluation RQ6 Validation results Corrections necessary? Project Yes No List of recommendations Improvement recommendati ons Evaluation of improvement recommendati ons Consider applicability to agile processes at SSC RQ7 Agile description Page 7

8 Appendix B Time plan Page 8

9 Page 9

10 Page 10

11 Page 11

12 Page 12