GAIA. GAIA Software Product Assurance Requirements for Subcontractors. Name and Function Date Signature 15/09/05 15/09/05 15/09/05 15/09/05 15/09/05

Transcription

1 Title Page : i Software Product Assurance Requirements for Subcontractors Name and Function Date Signature Prepared by D.MUNCH Prime Contractor SPA Manager 15/09/05 Verified by D.PERKINS E-SVM PA Manager D.HERBIN Prime Contractor PA Manager 15/09/05 15/09/05 Approved by Authorized by T. PEACOCK (Head of Quality AEU) JL LARRERE (Head of Quality AEF) R.STODDART E-SVM Project Manager V.POINSIGNON Prime Contractor Program Manager 15/09/05 15/09/05 15/09/05 15/09/05 Application authorized by ESA PA Manager Document type Nb WBS Keywords SPAR, Software, Quality, Requirements

2 Page : ii SUMMARY This document specifies the Software Product Assurance Requirements applicable for the development of on-board software in the frame of the Project.

3 Page : iii DOCUMENT CHANGE LOG Issue/ Revision Date Modification Nb Modified pages Observations 1/0 15/09/05 Initial issue for B2/C/D/E proposal

4 Page : iv PAGE ISSUE RECORD Issue of this document comprises the following pages at the issue shown Page Issue/ Rev. Page Issue/ Rev. Page Issue/ Rev. Page Issue/ Rev. Page Issue/ Rev. Page Issue/ Rev. ALL 1/0

5 Page : v TABLE OF CONTENTS 1 INTRODUCTION PURPOSE SUMMARY SCOPE REFERENCES APPLICABLE DOCUMENTS REFERENCE DOCUMENTS GLOSSARY AND ABBREVIATIONS TAILORED VERSION OF ECSS-Q-80B... 4 ANNEX 1 - SOFTWARE QUALITY MODEL ANNEX 2 - SOFTWARE METRICS... 22

6 Page : 1 1 INTRODUCTION 1.1 PURPOSE The purpose of this document is to specify the Software Product Assurance Requirements applicable for the development of on-board software products in the frame of the project. 1.2 SUMMARY This document contains the following chapters: Chapter 1 introduces the document, its purpose and scope Chapter 2 provides the list of applicable and reference documents Chapter 3 contains conventions in use in this document Chapter 4 contains the tailored version of ECSS-Q-80B 1.3 SCOPE This ECSS-Q-80B tailored version as well as the ECSS-E-40B standard are applicable to the on-board software products, developed in the frame of the project. The software engineering requirements are described in ECSS-E-40 Part 1B and Part 2B. The software configuration management requirements are described in ECSS-M-40B; complementary requirements are given in chapter of the ECSS-Q-80B tailored version. The risk management requirements are described in ECSS-M-00-03B; complementary requirements are given in chapter of the ECSS-Q-80B tailored version.

7 Page : 2 2 REFERENCES 2.1 APPLICABLE DOCUMENTS The following publications form a part of this document to the extent specified herein. Unless an issue is quoted for a document, the current issue is deemed to apply. When an issue is quoted, that issue and no other must be used. Title AD1 Product Assurance & Safety Requirements AD2 Mission Requirement Document AD3 Statement of Work AD4 Glossary of terms AD5 Space product assurance - Policy and Principles AD6 Space product assurance - Quality Assurance AD7 Space product assurance - Dependability AD8 Space product assurance - Safety AD9 Space product assurance - Software Product Assurance AD10 Space project management - Risk Management AD11 Space project management - Project phasing and planning Reference -EST-RD ASF.PLN.SAT EST-SOW ECSS-P-001 ECSS-Q-00 ECSS-Q-20B ECSS-Q-30B ECSS-Q-40B ECSS-Q-80B ECSS-M-00-03B ECSS-M-30A AD12 Space project management - Organization and conduct of reviews ECSS-M AD13 Space project management - Configuration management ECSS-M-40B AD14 Space engineering - Software - Part 1: Principles and requirements ECSS-E-40 Part 1B AD15 Space engineering - Software - Part 2: Document requirements definitions (DRDs) ECSS-E-40 Part 2B 2.2 REFERENCE DOCUMENTS The publications listed below were used in the preparation of this document, and contain background information relating to the subjects addressed. Title RD1 Guidelines and Standard for Software Product Assurance Requirements ADS.E.354 Reference

8 Page : 3 3 GLOSSARY AND ABBREVIATIONS The glossary of terms and abbreviations used in the current document are the one described in ECSS- Q-80B plus those identified here-after: A AD AwC AwN AwT CPU ISVV N/A RD REPL SCMP SVF SW TBC TBD V&V Term Signification Applicable Applicable Document Applicable with Comment Applicable with New requirement(s) Applicable with Tailoring Central Processing Unit Independent Software Verification and Validation Non Applicable Reference Document REPLaced Software Configuration Management Plan Software Validation Facility SoftWare To Be Confirmed To Be Defined Verification and Validation

9 Page : 4 4 TAILORED VERSION OF ECSS-Q-80B Here after tables define the tailoring of the following ECSS-Q-80B chapters: 5 - Software product assurance programme implementation 6 - Software process assurance 7 - Software product quality assurance These tables contain following information:: The applicability of each ECSS-Q-80B requirements (or set of requirements) is identified as follows: A: Applicable. NA: Not Applicable. REPL: Not Applicable and replaced by new requirement(s). AwN: AwT: AwC: Applicable with New additional requirement(s) to take into account Applicable with Tailoring. Applicable with Comment. For REPL or AwN requirements, new requirement(s) is(are) identified and described. For AwT or AwC requirements, the requirements tailoring or the comments are described.

10 Page : 5 5 Software product assurance programme implementation 5.1 Introduction 5.2 Organization and responsibility Requirement 5.2.1, 5.2.2, A Software product assurance manager Requirement , A Training Requirement , , , AwN The following requirement is added: A The supplier shall demonstrate in its proposal that personnel with appropriate skills will be available for the project. 5.3 Contractual aspects Requirement 5.3 A 5.4 Software product assurance programme management Software product assurance planning and control Requirement , , A Requirement AwT The software product assurance plan shall also specify or reference the following items: The criticality of the software (refer to subclause ). Software quality verifications that will be performed on the software product. For each output from development phases (see subclause 6.1.5) following information shall be given: - Type of verification performed, - Standard used, - Tool used (if any), - Control schedule, - Control range (exhaustive or by sampling). When the control is performed by sampling, the control percentage and the selection criteria shall be given Software quality verifications that will be performed on the software processes. For each process following information shall be given: - Type of control performed, - Procedure used, - Control schedule. Following processes shall be considered: - SW management process - SW acquisition process - SW engineering processes - Infrastructure process - SW support processes (documentation, configuration management, V&V, reviews,

11 Page : 6 audits) - Subcontractors management process Product and process metrics to be collected, recorded, analysed and reported (see subclauses for process metrics and subclauses for product metrics). The Compliance Matrix that shall document the compliance of the supplier s dispositions with the present document and with ECSS-E-40 part 1B and part 2B Requirement , A Software product assurance reporting Requirement , , , AwN The 2 following requirements are added: A In addition to SW products and SW process assessment (refer to sub-clauses and ), the Software product assurance report shall include following information: SW quality activities performed, Results of SW quality verifications, SW Product metrics (see subclause 6.2.5), SW Process metrics (see subclause 7.1.4), Necessary actions identified from product and process assessments, Main identified risks and related actions. EXPECTED OUTPUT: Software product assurance report [PAF; SRR, PDR, CDR, QR, AR, ORR] B The evolution of information included in the Software product assurance report shall be reported monthly. EXPECTED OUTPUT: Periodic software product assurance report [monthly] Audits Requirement AwN The following requirement is added: A For each performed audit, an audit report shall be issued and distributed to all parties, including customer. Audits actions and recommendations shall be followed up to close out. EXPECTED OUTPUT: Audit report Alerts Requirement A Nonconformances Requirement , , A Software problems Requirement , , A 5.5 Risk management and critical item control Risk management

12 Page : 7 Requirement AwN The 4 following requirements are added: A A risk analysis shall be performed before the first issue of the Software development plan and results integrated in the plan B The Software product assurance shall take part in risk identification and risk analysis C The risk analysis shall be updated on a monthly basis. EXPECTED OUTPUT: Risk Assessment Report D Identified risks shall be regularly reported to the customer. EXPECTED OUTPUT: Risk Assessment Report Critical item control Requirement A 5.6 Supplier selection and control Supplier selection Requirement AwN The following requirement is added: A Pre-award audits shall be performed on subcontractors who are not known on a regular working basis by the supplier or who have presented difficulties during previous contracts. The monitoring of subcontractors shall be adjusted, based on the results of these audits. NOTE In addition to ISO standard, due consideration shall be given to third party assessments against maturity models such as ISO 15504, S4S, CMMi, EXPECTED OUTPUT: Audit report Supplier requirements Requirement AwT The following text is added to the requirement: The present document shall be the basis for establishing the software product assurance requirements for the next level suppliers. Requirement A Supplier monitoring Requirement , , , A Criticality classification Requirement A 5.7 Procurement Requirement 5.7.1, 5.7.2, 5.7.3, 5.7.4, 5.7.5, 5.7.6, A

13 5.8 Tools and supporting environment Page : 8 Requirement 5.8.1, A Methods and tools Requirement AwT The following text is added to the requirement: Tools shall be used for the following activities: Code complexity analysis, Code standard analysis, Code coverage analysis, Traceability. Requirement , , , , A Tool selection Requirement , A 5.9 Assessment and improvement process Requirement AwT The reference to ECSS-Q is deleted as defined in [AD1]. Requirement 5.9.2, 5.9.3, 5.9.4, 5.9.5, 5.9.6, A

14 Page : 9 6 Software process assurance 6.1 Software development life cycle Requirement AwC The following comment is added to the requirement: NOTE ECSS-E-40B is applicable and fully describes requirements for software life cycle. Requirement 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.8, A Requirement AwT The following text is added to the requirement: The data packages produced for each milestone shall be described in SW plans. Data packages shall be delivered to the customer, at least 2 weeks before the milestone. NOTE Milestones identified in ECSS-E-40B (SRR, PDR, CDR, QR, AR) and associated requirements are applicable. 6.2 Requirements applicable to all software engineering processes Documentation of processes Requirement , , , , , , , AwN The 2 following requirements are added: A Plans standards and procedures shall be provided for review by the customer in the data package of the review planned before the phase for which they are applicable. EXPECTED OUTPUT: Procedures and standards [PAF;SRR, PDR, CDR] B Each standard shall include: A set of writing rules and a template of related document, Adequate rules to enforce the selected characteristics (see sub-clause 7.1.3). EXPECTED OUTPUT: Procedures and standards [PAF; PDR]. Requirement REPL The requirement is replaced by the following one: A The software product assurance plan shall identify all plans, standards and procedures to be produced and used, the relationship between them and the time-scales for their preparation and update. EXPECTED OUTPUT: Identification of software plans, their interrelations and schedule for preparation [PAF; SRR, PDR]. Requirement AwT The following text is added to the requirement: The following activities shall also be covered by development procedures and projects standards: Requirements production and documentation, Design production and documentation, Unit tests, Integration tests,

15 Validation tests, SW project management planning and tracking procedure, SW margins estimates. EXPECTED OUTPUT: Procedures and standards [PAF; PDR] Software dependability and safety analysis Page : 10 Requirement AwT The following text is added to the requirement, as defined in [AD1]: Critical software shall be identified in the Unit Requirement Specification. All activities for critical software shall be defined in the Software PA Plan. Requirement , , , , , A Handling of critical software Requirement AwT The reference to ECSS-Q is deleted as defined in [AD1]. Requirement , , , , , , , A Software configuration management Requirement , , , , , , , , The 3 following requirements are added: AwN A A computer-based configuration management tool shall be used. This tool shall manage configuration items, Software Problem Reports, Software Change Request, Software Waivers and Deviations. The tool shall be identified in the Configuration Management Plan. EXPECTED OUTPUT: Software configuration management [MGT; SRR, PDR] B For each output from development phases (see subclause 6.1.5) the configuration date shall be defined in the SW Configuration Management Plan. EXPECTED OUTPUT: Software configuration management [MGT; SRR, PDR] C Each SW product delivery shall be accompanied by a SW Release Document (refer to DRD given in Annex D of ECSS-E-40 part 2B) and associated Software Configuration File (refer to DRD given in Annex F of ECSS-M-40B) EXPECTED OUTPUT: SW Release Document, SW Configuration File. Requirement AwT "ECSS-M-40" is replaced by "ECSS-M-40B" in the text of the requirement. The following note is added to the requirement: NOTE: As requested in ECSS-M-40B Subclause 5.2.1, the supplier shall develop a SW Configuration Management Plan The SW Configuration Management Plan shall be provided with the SW Product Assurance Plan for approval by the customer. Requirement AwT

16 The following text is added to the requirement: Page : 11 If the protection mechanism used is based on a specific tool, this tool shall be supplied to the customer along with the delivered software products. However, selection of tools that are available publicly should be preferred. Requirement AwT The following text is added to the requirement: The software identifier shall be included in the SW itself (hard coded) Process metrics Requirement , , , A Requirement , REPL These requirements are replaced by the following one: A Process metrics identified in ANNEX 2 shall be used within the supplier s organisation and reported to the customer. EXPECTED OUTPUT: Process metrics specification and justification in the software product assurance plan [PAF; SRR, PDR]. Metrics reports in software product assurance reports [PAF; PDR, CDR, QR, AR, ORR] Verification Requirement , , , , , , , , , , , Requirement AwT The following text is added to the requirement, as defined in [AD1]: The list of software's considered "Highly critical software" shall be established by the Prime Contractor and submitted to ESA for approval Reuse of existing software Requirement , , , , , , , A Requirement AwT The choice of reused SW shall also take into account: The availability and competency of maintenance personnel, The available margins (memory, CPU and others critical computer resources). Requirement AwT The following aspects shall also be assessed: The compatibility between the quality rules applied during the development of the reused SW and the requirement defined in the present document, The availability of documentation requested in this document, The open actions on the reused SW. Requirement AwC A The following comment is added to the requirement: NOTE: The SRF DRD is given in Annex I of ECSS E40 Part 2B 6.3 Requirements applicable to individual software engineering processes or activities

17 6.3.1 Software requirements analysis Page : 12 Requirement , , , , A Software architectural design and design of software items Requirement , , , , , , , , The following requirement is added: AwN A The Software Design Document shall describe hierarchy, dependency and interfaces of SW components. It shall also describe dynamic aspects and data s control flow. NOTE: Refer to SDD DRD given in Annex C of ECSS E40 Part 2B Coding Requirement , , , , , , , A Requirement REPL The requirements are replaced by the following one: A The SW shall be developed using a high level language except where explicitly exempted. The use of a non high level coding language (e.g. assembler) shall be shown to be absolutely necessary to achieve the required performances. EXPECTED OUTPUT: Document justifying suitability of the language [DJF; PDR]. Requirement AwT NOTE 2 is suppressed (a tool to measure adherence of the code to the coding standards shall be used see amended subclause ) Testing and validation Requirement , , , , , , , , , , , , , , , , , , , , , , , , , , , Requirement AwT The following text is added to the requirement: 100% code branches and decisions coverage is required. 100% requirements (requirements baseline and technical specification) coverage is required. Requirement AwT The following text is added to the requirement: The following elements shall be reviewed during the TRR: Status of the Software Problem Reports, Software Change Request, Software Waivers and Deviations. Status of documentation, Readiness of validation test plan, procedures and test data. Resources (human resources, logistic, test tools). Schedule of the tests. The TRR shall conclude to the authorisation to start the tests session, if no previous points presented during the TRR is considered as blocking. Requirement AwT A

18 The following text is added to the requirement: Page : 13 The Test Review Board shall review the product configuration after the test phase, review and approve the test results obtained and identify further work to be done (if any) for release of the product for further phase. Requirement AwT The following text is added to the requirement, as defined in [AD1]: The list of software's considered "Highly critical software" shall be established by the Prime Contractor and submitted to ESA for approval Software delivery and acceptance Requirement , , , , , , , , Operations Requirement , , A Maintenance Requirement , , , A A

19 Page : 14 7 Software product quality assurance Requirement 7.1.1, A Quality models Requirement , , REPL These 3 requirements are replaced by the 2 following ones: A The Quality model defined in ANNEX 1 shall be used to specify the quality requirements. This model shall be tailored according to the specificities of the software to develop. The following minimum set of characteristics and subcharacteristics shall be taken into account: Characteristics Subcharacteristics Functionality Reliability Efficiency Maintainability Suitability, Accuracy, Interoperability, Functionality compliance. Maturity, Fault tolerance, Recoverability, Reliability compliance. Time behaviour, Resource utilisation, Efficiency compliance. Analysability, Changeability, Stability, Testability, Maintainability compliance. Sw Development Efficiency Process Efficiency. Simplicity, Modularity, Self Descriptiveness, Traceability, Conciseness. Efficiency can be contradictory with maintainability and reliability characteristics. So other characteristics have to be preferred against efficiency, except when actual constraints on CPU or memory budgets exist. EXPECTED OUTPUT: Software quality models [PAF; PDR] B The characteristics and subcharacteristics of the quality model shall drive the quality assurance activities to be performed. The standards shall include adequate rules to enforce the selected characteristics compliance. Associated quality controls shall be performed. EXPECTED OUTPUT: Characteristics and subcharacteristics reflected in standards [PAF;SRR, PDR, CDR]. Assessment of the characteristics of the software product in the software product assurance report [PAF; SRR, PDR, CDR, QR,AR, ORR] Product metrics Requirement A Requirement AwT The reference to ECSS-Q is deleted as defined in [AD1].

20 Chapters: and Page : 15 Requirement 7.1.5, 7.1.6, A Chapters: Basic Metrics Requirement REPL The requirement are replaced by the following one: A Products metrics identified in ANNEX 2 shall be used within the supplier s organisation and reported to the customer. EXPECTED OUTPUT: Product metrics specification and justification in the software product assurance plan [PAF; SRR, PDR]. Metrics reports in software product assurance reports [PAF; PDR, CDR, QR, AR, ORR]. Chapters: to Requirement 7.1.8, 7.1.9, , , , A 7.2 Product quality requirements Technical specification Requirement , , , A Design and related documentation Requirement , , A Software intended for reuse Requirement , , , , , , A 7.3 Supporting documentation Test and validation documentation Requirement , , , , , A Reports and analysis Requirement A 7.4 Standard hardware for operational system 7.4.1, 7.4.2, 7.4.3, Firmware Requirement 7.5.1, 7.5.2, AwN The 4 following requirements, specific to firmware embedded in ASIC and FPGA are added: 7.5.A VHDL coding standards shall be specified and observed. EXPECTED OUTPUT: VHDL Coding standards [PAF; PDR].

21 Page : B 100 % coverage of VHDL code shall be achieved. EXPECTED OUTPUT: Collected data and analysis of the results in the software product assurance report [PAF; CDR, QR, AR]. 7.5.C Following product assurance activities, related to firmware resident in ASIC or FPGA shall be performed: Analysis of the Development Plan, the Verification Plan and the Design Validation Plan, in order to check their conformity to the applicable requirements; Verification of the actual implementation of these plans, by means of inspections, reviews, and audits if necessary; Verification of the VHDL coding conformity to the applicable coding rules; Verification of the traceability between the test plans/procedures and the Requirements Specification; Verification of the code coverage rate achieved; Verification that the timing of critical signals are properly checked and documented; Assessment of the quality and of the completeness of the provided documentation; Attendance to the following reviews: SRR, PDR, DDR, CDR, QR/AR; Check that all inputs to the design, that are not automatically generated and are necessary to reproduce the design such as simulation pattern, schematics, VHDL source codes, synthesis scripts, are put under configuration control; Verification that the configuration management system is properly defined and used. 7.5.D Monthly software product assurance reports and software product assurance reports for reviews shall be produced. NOTE: For reports contents, refer to subclauses 5.4.2A and 5.4.2B. EXPECTED OUTPUT: Software product assurance report [PAF; SRR, PDR, CDR, QR, AR, ORR]. Periodic software product assurance report [monthly].

22 ANNEX 1 - SOFTWARE QUALITY MODEL Page : 17 The software quality objectives that are applied to guide the development of software products are expressed as a set of software quality characteristics. These characteristics are further subdivided into subcharacteristics, which may be measured by a set of metrics. The quality model presented herein is a tailored version of the general quality model defined in IS The table below defines each software product characteristic and subcharacteristic (as defined in ISO 9126). Characteristic Subcharacteristic Subcharacteristic Description Functionality The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions. Suitability Accuracy Interoperability Security Functionality compliance The capability of the software product to provide an appropriate set of functions for specified tasks and user objectives. The capability of the software product to provide the right or agreed results or effect with the needed degree of precision. The capability of the software product to interact with one or more specified systems. The capability of the software product to protect information and data so that unauthorised persons or systems cannot read or modify them and authorised persons or systems are not denied access to them. The capability of the software product to adhere to standards, conventions or regulations in law and similar prescriptions relating to functionality.

23 Page : 18 Characteristic Subcharacteristic Subcharacteristic Description Reliability The capability of the software product to maintain a specified level of performance when used under specified conditions. Usability The capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions. Maturity Fault tolerance Recoverability Reliability compliance Understandability Learnability Operability Attractiveness Usability compliance The capability of the software product to avoid failure as a result of fault in the software. The capability of the software product to maintain a specified level of performance in cases of software faults or of infringement of its specified interface. The capability of the software product to re-establish a specified level of performance and recover the data directly affected in the case of a failure. The capability of the software product to adhere to standards, conventions or regulations relating to reliability. The capability of the software product to enable the user to understand whether the software is suitable, and how it can be used for particular tasks and conditions of use. The capability of the software product to enable the user to learn its application. The capability of the software product to enable the user to operate and control it. The capability of the software product to be attractive to the user. The capability of the software product to adhere to standards, conventions, style guide or regulations relating to usability.

24 Page : 19 Characteristic Subcharacteristic Subcharacteristic Description Efficiency The capability of the software product to provide appropriate performances, relative to the amount of resources used, under stated conditions. Maintainability The capability of the software product to be modified. Modification may include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specifications. Time behaviour Resource utilisation Efficiency compliance Analysability Changeability Stability Testability Maintainability compliance The capability of the software product to provide appropriate response and processing times and throughput rates when performing its function, under stated conditions. The capability of the software product to use appropriate amount and types of resources when the software performs its function under stated conditions. The capability of the software product to adhere to standards or conventions relating to efficiency. The capability of the software product to be diagnosed for deficiencies or causes of failures in the software, or for the parts to be modified to be identified. The capability of the software product to enable a specified modification to be implemented. The capability of the software product to avoid unexpected effects from modifications of the software. The capability of the software product to enable modified software to be validated. The capability of the software product to adhere to standards or conventions relating to maintainability.

25 Page : 20 Characteristic Subcharacteristic Subcharacteristic Description Portability The capability of the software product to be transferred from one environment to another. Adaptability Installability Co-existence Replaceability Portability compliance The capability of the software product to be adapted for different specified environments without applying actions or means other than those provided for this purpose for the software considered. The capability of the software product to be installed in a specified environment. The capability of the software product to co-exist with other independent software in a common environment sharing common resources. The capability of the software product to be used in place of another specified software product for the same purpose in the same environment. The capability of the software product to adhere to standards or conventions relating to portability. In addition to ISO IS0 9126, following product subcharacteristics are also considered. They influence several of the previously defined characteristics. Subcharacteristic Subcharacteristic Description Main Influenced Characteristics Simplicity Modularity Self Descriptiveness Traceability Conciseness Attribute of the software that provides implementation of functions in the most understandable manner (avoidance of practices which increase complexity). Attribute of the software that provides a structure of highly independent modules. Attribute of the software that provides explanation of the implementation of functions. Attribute of the software that provides a thread from the requirements to the implementation. Attribute of the software that provide for the implementation of functions with a minimum amount of code. Maintainability Reliability Maintainability Portability Maintainability Portability Functionality Maintainability

26 Page : 21 Furthermore, in addition to ISO IS0 9126, the table below defines SW process characteristics and subcharacteristics: Characteristic Subcharacteristic Subcharacteristic Description SW Development Effectiveness The degree of success/quality of the software development process to which the software has been subjected, which provides valuable indications of the product quality. Process Effectiveness Evidence that shows which activities have been performed during the development process to adhere to standards, conventions and regulations considered relevant for the evaluation of the software quality.

27 Page : 22 ANNEX 2 - SOFTWARE METRICS The quality characteristics and subcharacteristics can be evaluated during the software life cycle, through internal and external metrics. - Internal metrics, applied to non-executable software product during its development stages of the software life cycle. They allow to measure the quality of intermediate products elements and thereby to predict the quality of the final product. - External metrics, applied on the executable software product. They can only be used during the testing stages of the software life cycle. They allow to measure the quality of the software product behaviour in the test environment. The set of metrics that shall be collected is defined hereafter. It addresses both process metrics (related to process effectiveness characteristics) and product metrics (related to software product characteristics).

28 Page : 23 Internal metrics Metric Metric limit Associated main [Subchracteristics] Characteristics Requirements stability: Ratio between number of changed requirements (added or modified since the first version of the requirements baseline) and total number of requirements defined in the first version of the requirements baseline. Number of open and closed change requests. Number of open and closed non-conformances. Number of open and closed requests for waiver. Actual phase durations and milestones dates versus initially planned ones. Actual effort consumed per phase versus initially estimated ones. Processor utilisation: CPU load according to worst cases defined scenarios (evaluated then measured). No limit defined (indicator) The ratio shall be stabilized in late phases. No limit defined (indicator) The nb of open CR shall converge to 0 in late phases. No limit defined (indicator) The nb of open NC shall converge to 0 in late phases. No limit defined (indicator) The nb of open RFW shall converge to 0 in late phases. No limit defined (indicator) No limit defined (indicator) Limit to be defined according to project requirements [Suitability] Functionality [Suitability] Functionality [Process Effectiveness] Sw Dev. effectiveness [Suitability] Functionality [Maturity] Reliability [Process Effectiveness] Sw Dev. effectiveness [Suitability] Functionality [Process Effectiveness] Sw Dev. effectiveness [Process Effectiveness] Sw Dev. effectiveness [Time behaviour] Efficiency

29 Page : 24 Metric Metric limit Associated main [Subchracteristics] Characteristics Ratio between the memory size used and the total memory size available physical and logical (evaluated then measured). Ratio between number of branches covered by tests and the total number of branches (per module). Ratio between number of single decisions covered by tests and the total number of single decisions (per module). Ratio between number of performed test and the total number of planned tests (per tests categories) Total number of open and closed problems detected during software elements (documents, code) inspections (including software product assurance controls). Total number of open and closed actions (including actions initiated by software product assurance and reviews actions). McCabe cyclomatic number per module. Limit to be defined according to project requirements =100% =100% [Resource utilisation] Efficiency [Testability] Maintainability =100% [Process Effectiveness] Sw Dev. effectiveness No limit defined (indicator) The nb of open problems shall converge to 0 in late phases. No limit defined (indicator) The nb of open actions shall converge to 0 in late phases 15 [Suitability] Functionality [Maturity] Reliability [Process Effectiveness] Sw Dev. effectiveness (*) All [Simplicity, Changeability, Analysability] Maintainability Reliability Number of source code instructions per module. 100 [Conciseness] Maintainability

30 Number of nested control structures per module. Page : 25 Metric Metric limit Associated main [Subchracteristics] Characteristics Average number of modules per level (number of components divided by number of levels). Average number of calls per module (number of calls between modules divided by the number of modules) Comment rate per module: Ratio between number of comments lines and total number of lines in the module. Ratio between number of requirements traced in the design and code and total number of requirements defined in technical specification. (*) This metric is also related to the compliance subcharacteristic of all characteristics % = 100% [Simplicity, Changeability, Analysability] Maintainability Reliability [Modularity, Changeability, Analysability] Maintainability Portability [Modularity, Changeability, Analysability] Maintainability Portability [Self descriptiveness, Changeability, Analysability] Maintainability Portability [Traceability] Functionality

31 External metrics Page : 26 Metric Definition Metric limit Associated main [Subchracteristics] Characteristics Ratio between number of correctly implemented requirements confirmed by tests, analysis or inspections and total number of requirements defined in requirements baseline. = 100% [Suitability] Functionality Ratio between number of correctly implemented requirements confirmed by tests and total number of requirements defined in requirements baseline. Ratio between number of correctly implemented requirements confirmed by test, analysis or inspections and total number of requirements defined in technical specification. Ratio between number of correctly implemented requirements confirmed by test and total number of requirements defined in technical specification. Ratio between number of detected non-conformances and total number of executable statements (in KLoC). Number of non-conformances discovered during software integration and validation. Number of non-conformances discovered on SW products delivered by the subcontractor. Those delivered SW products being used at upper level (equipment, subsystem, system) for integration, validation, operation purposes. 90% = 100% 90% No limit defined (indicator) No limit defined (indicator) No limit defined (indicator) [Suitability] Functionality Reliability

32 Page : 27 DISTRIBUTION LIST Overall document Action Information Summary