Validation of Automotive Software

Transcription

1 Independent Verification and Validation of Automotive Software Independent Verification and Validation for Automotive Software

2

3 Automotive Software IV&V.0 The automotive industry spends approximately 2 to 3 billion per year fixing software problems. For decades software has played a critical role in the aeronautics, space, railway and defense industries. Independent Verification and Validation (IV&V) is mandatory practice in these industries, where reliability is of paramount importance. The direct return on investment (ROI) for Software IV&V has been calculated for many projects. These calculations prove that software IV&V, seen just from a cost-benefit point of view, yields ROI factors in the order of 1.5 to 10. The investment in software IV&V typically corresponds to about 10% of the investment in the software development. Software plays an ever more critical role in the automotive industry. Software errors can result in expensive vehicle recalls, a damaged reputation and even the loss of human life. There is a growing need to adopt Software IV&V in the automotive industry. Rovsing is proud of its world-class reputation for developing, verifying and validating safety-critical, embedded software for the space industry and it is now bringing its experience in Software Independent Verification and Validation to the automotive industry.

4 Automotive Software.0 The automotive industry is experiencing a dramatic transformation, a transformation not about steel and rubber but about electronics and software. Today it is widely acknowledged that 80% to 90% of vehicle innovations are based on software systems, and these systems can account for up to 30% or more of a vehicle s manufacturing cost. Software has become an indispensable core technology for the automotive industry. Current trends indicate that every four years there is a tenfold increase in the volume of software in high-end vehicles. 1 GB 100 MB 10 MB 1 MB 100 KB Every 4 year: tenfold increase in software content 10 KB Software is key to the automotive industry s response to environmental and legislative pressure to: significantly reduce traffic accidents and fatalities achieve efficient and sustainable transport solutions deliver an enhanced driver experience increase throughput through real-time traffic management reduce fuel consumption and increase economy reduce emissions Currently 50% to 70% of the development costs for an ECU are related to software IV&V = Software Quality = Core Technology

5 .05 Many high-end vehicles today contain more than 70 distributed microcontrollers, 100 MegaBytes of code, 5 or more distinct communication networks, and have several thousand data and control signals being exchanged in real-time every second. In-vehicle networks such as CAN, LIN, FlexRay TM and MOST TM connect distributed sensors, actuators, displays and processors; fusing subsystems into systems with complex functionality. Today, if you change a line of code, you re looking at the potential for some major problems. Hardware is very predictable, very repeatable. Software is in much more of a transient state William Powers, VP of research at Ford. Automotive software is often on the critical development path and as such is subject to the antagonistic pressures of meeting delivery deadlines and achieving the necessary software quality levels. However, in order to achieve future strategic technological and business goals the automotive industry will rely heavily on high quality software. The investment in software has become a higher percentage of the total vehicle manufacturing cost. This situation has generated a significant interest in practices and technologies which improve quality, reduce development cost, and mitigate the risks associated with software intensive systems. One such practice is Independent Verification and Validation (IV&V), which is now mandatory in the space, defense, aviation, railway and nuclear industries. Rovsing is a recognised world leader in the provision of IV&V services. IV&V = Software Quality = Core Technology

6 .0 Independent Software Verification and Validation Even the most experienced programmers can produce code that contains errors. Studies of software developed for a wide range of applications give a conservative estimate for the number of residual errors at 1 error per 1,000 lines of normal code and 1 per 10,000 lines for critical software. This figure is independent of the programming language used. Where software failure can lead to the loss of life, catastrophic or expensive failures, or significant damage to a supplier s reputation, Independent Verification and Validation (IV&V), is used to minimise the exposure to such risks. Benefits of IV&V Higher confidence in the software safety and dependability Reduced development risk, time and cost with earlier detection of errors Conformance to requirements fewer discrepancies between requirements and product Better decision criteria through clearer visibility into the development process Higher reliability and robustness of the product resulting from independent stress testing Increased usability and maintainability with higher quality and consistency of documentation If another person tells me how to make a million I will scream, but if someone tells me how to save a dollar I will stay up all night thinking about it Henry Ford founder of the Ford Motor Company IV&V is employed as part of a software quality assurance program and involves the Verification and Validation of a software product by an organisation which is technically, financially, and managerially separate from the organisation responsible for the product development. IV&V is a mandatory practice for the certification of safety-critical software in the space, defense, aviation, railway and nuclear industries. Independent software validation differs significantly from non-independent software testing. The goal of non-independent software testing is to prove that the software works. The goal of independent software validation is to expose residual errors. IV&V = Software Quality = Return on Investment

7 .07 Principles of IV&V Separation of concerns Independent and complementary to the developer s V&V Customer defines the scope The customer decides on the scope and budget Earlier is better The sooner IV&V is involved in the project the better Reasonably mature product Documentation and code is submitted to IV&V when in reasonably mature condition IV&V is process flexible The IV&V process is adapted to the software development process Return on Investment The greater the potential cost of a software failure, then the larger the return of investment for IV&V Experienced people IV&V is conducted by personnel experienced in software development, test and quality assurance The Return On Investment for IV&V is typically between 1.5 and 10 times the investment 1 Importance of Independence Self-verification and self-validation can be ineffective IV&V increases confidence in the process Self-certification of safety-critical systems may leave residual errors IV&V actively seeks to find residual errors Technical, managerial and financial independence are necessary to ensure no conflicts of interest are present IV&V helps to ensure quality, protects your customer, your business, your reputation and your brand. 1. Source: Direct Return on Investment of Software Independent Verification and Validation: Methodology and Initial Case Studies, J.B. Dabney & G. Barber, Assurance Technology Symposium, 5 June IV&V = Software Quality = Return on Investment

8 Why IV&V?.0 Can the automotive industry benefit from the experience of the space industry? Automotive software related recalls include: 62,369 vehicles in 2007: the antilock brake system (ABS) control module software caused the rear brakes to lock up during certain braking conditions. This error resulted in a loss of vehicle control causing a crash without warning. The ABS electronic control unit was reprogrammed. 5,902 vehicles in 2006: under low battery voltage condition the air bag control unit improperly sets a fault code and deactivates the passenger side frontal air bag. The airbag subsequently would not deploy in the event of a collision. Dealers updated the air bag control module software. 127,928 vehicles in 2006: the fuel pump module and the power train control module (PCM) software allowed the engine to stall under certain operating conditions. Dealers reprogrammed the power train control module. 2,333 vehicles in 2006: the software programmed into the powertrain control module caused a momentary lock up of the drive wheels if the vehicle was traveling over 40 mph and the operator shifted from drive to neutral and back to drive. The powertrain control module was reprogrammed. 160,000 vehicles in 2005: a software fault caused the hybrid vehicle s gasoline engine to unexpectedly shut down. 68,000 vehicles in 2004: a system failure that could inadvertently select reverse gear if the controller detected a major loss of transmission-oil pressure. 3,500,000 vehicles in 1999: a software defect resulted in vehicle stopping distances being extended by 15 to 20 meters. Investigators received almost 11,000 complaints as well as reports of 2,111 crashes and 293 injuries. IV&V = Software Quality = Revenue

9 . For decades the space industry has developed software to the highest possible standards for some of the most challenging applications mankind has seen. IV&V has evolved out of such endeavors to be the recognised state-ofthe-art software engineering practice to ensure correctness in an industry where reliability is paramount. The software engineering standards most relevant to the automotive industry today such as CMMI, IEC-15504, IEC and ISO require the use of Independent Software Verification and Validation. IV&V Technical Benefits IV&V Management Benefits Improved software/system performance Clearer visibility into development Higher confidence in software reliability Enhanced decision criteria Conformance between specification and code Provides a second source technical alternative Criteria for program acceptance Lower development cost Independent technical expertise Reduced maintenance cost It is conservatively estimated that the automotive industry spends 2 billion to 3 billion per year fixing software problems IV&V = Software Quality = Revenue

10 Rovsing and IV&V.10 Rovsing is a recognised leader in the provision of IV&V services. We partner with our customers to achieve on-time, on-budget results of the highest possible standards. Through consultation Rovsing helps to establish the customer requirements and tailor the IV&V service to support the customer s strategic business goals, going far beyond a one-size-fits-all approach. Rovsing strives to ensure that its customers reap the rewards of their IV&V investment overand-over again through good system design, good software design, scalable architecture, platform design and software component reuse. World Class Expertise Rovsing has recently completed IV&V for the European Space Agency s Automated Transfer Vehicle (ATV). The ATV is the largest and most sophisticated automated unmanned space vehicle ever built, containing on-board software of nearly one million lines of code. This 20 tonne supply spacecraft for the International Space Station (ISS) delivers essential fuel, gasses, scientific equipment, food and other life support necessities. Docking between the ATV and ISS is performed completely under computer control, making the ATV the first unmanned space vehicle to perform such critical maneuvers autonomously. Rovsing has an established track record as a world class supplier of effective and efficient IV&V services The MSU (ATV) software has been designed and developed within a tight collaboration between the ESA and EADS-ST technical teams, gathered in the so-called MSU team which consists of a dozen flight control and software specialists. From the early development stages, ROVSING has been involved in the development and test campaign of the MSU software, providing feedback on the results of their own independent assessment to the MSU team. This successful collaboration started in 2001, and the qualification of the Category A software was achieved in March This development is a premiere in Europe, and is, above all, the result of the successful day-to-day collaboration Eric Zekri, European Space Agency IV&V of the ATV software has earned Rovsing the prestigious position of being the first European company to successfully apply IV&V to category A on-board software for spaceflight. Category A classifies the most critical of software systems. IV&V = Software Quality = Your Reputation

11 .11 IV&V Methods Rovsing has developed unique IV&V methodes, such as Critical Areas Identification Method (CAIM), for criticality analysis. CAIM is a powerful method to identify aspects of a software design and code segments which are potential common mode failure points and thus require the highest levels of attention and risk mitigation. The CAIM process is used to guide the subsequent verification and validation activities. Verification of Automatically Generated Code Model-Based Software Development methods which utilise Automatic Code Generation are in widespread use in the automotive, aeronautical, space and defense industries. Rovsing is a pioneer in providing IV&V services for projects utilising Model-Based Software Development Methods and Automatic Code Generation and has developed its own unique assessment process. Rovsing IV&V helps bring you the quality and value you want, on time, within budget, every time As an independent organisation Rovsing is not associated with any company in the automotive industry. Rovsing is proud of its proven reputation developing safety-critical systems for the space industry and providing IV&V services for the most demanding and safety-critical of applications. Critical Requirements Level documentation review Excluded by design Critical Design related Architectural & Detailed Design Level documentation review Critical Excluded by implementation Implementation related Source code review IV&V - Verification Concept by CAIM method (Critical Area Identification Method developed by Rovsing) IV&V = Software Quality = Your Reputation

12 IV&V for Automotive Software.12 Unprecedented growth in the sophistication and complexity of embedded systems in vehicles is not matched by advances in the degree of certification, reliability assurance, verification and validation. Thus, with each progression in technology, the risk of software errors remaining undetected in the process of design, verification and validation using low-cost empirical methods increases substantially. The aerospace, defense, railway and nuclear industries have realised this for some time and consequently employ IV&V as an effective countermeasure to the risks of software failure. The pervasiveness of electronic systems in vehicles now calls for the largest paradigm shift in automotive development since the invention of the car The notion that bugs in the software are avoidable is mistaken, as far as I m concerned. They aren t Stephen Wolfsried, head of Electrical/Electronic system and chassis unit at DaimlerChrysler s Mercedes Car Group. See: OEM Requirement Definition Verification / Evaluation Improve Request Vehicle Test Calibration System Design Validation Improve Request System Verification Hardware in the Loop Tests SubSystem Design Validation Improve Request Module Test Code Generation C-Code ECU Supplier s Realm IV&V = Software Quality = Strategic Business

13 .13 Initiatives such as OSEK and AUTOSAR aim to standardise automotive software architecture in an effort to increase quality and reduce development costs. Complementary to this, several of the software engineering standards applied within the automotive industry, such as CMMI, IEC-15504, IEC and ISO require the use of Software Independent Verification and Validation as an effective practice to increase the final quality of software systems. Over 50% of vehicle warranty costs are caused by electronics and software. The industry needs increased up-front validation and testing efforts, enhanced change control and program management processes Automotive Engineering 2010, achieving more from less. Detroit, July, 2004 Roland Berger Strategy consultants. IV&V is proven to increase the reliability, robustness and overall quality of software systems by removing software errors at every stage of the software development process. Rovsing s IV&V services are designed to be applied in every phase of the software development process in order to increase quality and maximize the Return on Investment IV&V = Software Quality = Strategic Business

14 Rovsing Software IV&V services.14 Rovsing is an established provider of IV&V services for safety-critical software applications, conforming to the highest quality and software engineering standards. Rovsing provides a flexible and tailored service to meet its customer s requirements. Services provided include: Criticality Analysis Rovsing s Critical Areas Identification Method (CAIM) uses Software Failure Mode Effects and Criticality Analysis (SFMECA) on a conceptual model of the software and its surroundings in order to identify the failure modes of each of the components and analyse their effects. All effects are classified as either critical or non-critical. If a failure mode has any potentially critical effect then the software component, in which the mode occurs, is considered a critical area. This process guides the IV&V effort and ensures that the focus is on the appropriate areas. Supply of Software Validation Facilities A Software Validation Facility (SVF) is a custom designed test platform for ECUs, which allows testing of the software in real-time operational mode and in a fully representative environment. The SVF far exceeds the capability of standard commercially available test tools and facilitates normal and abnormal operating conditions, such as memory errors, bit flips, sensor failure, actuator failures, communication failures, buffer overloads, interrupt inter arrival errors, network overloads, missed deadlines, scheduling failures, abnormal event sequences and unexpected resets. The SVF automates both the execution of test scenarios and the capturing of results. Verification activities The verification process ensures that the software has been built correctly to the requirements and according to the relevant development process requirements e.g. IEC Verification involves detailed analysis of the software specification, design, algorithm implementation, code, unit and integration testing. The verification process helps to ensure that the software is complete, consistent and correctly implemented. This process is conducted in parallel with system development. Documentation, code, test procedures and results are analysed as soon as they are reasonably mature, and as early as possible in order to capitalise on cost benefits of early error detection. Validation activities Normal validation performed by developers aims to ensure that the correct software has been built and that this software performs correctly within the system context. However, independent validation testing also known as stress testing seeks specifically to expose errors in the software. This is to assure that the software performs to the customer s expectations under all operational conditions and no unexpected behaviour occurs when the software is subjected to extreme conditions and unanticipated events. This IV&V activity is performed using a dedicated software validation facility. IV&V = Rovsing = Your Development Partner

15 About Rovsing.15 Founded in 1992, and listed on the OMX Nordic stock exchange, Rovsing s highly skilled engineers and scientists provide services and support for the automotive, defense and space industries. Based in Copenhagen, Denmark, with subsidiaries in the USA and Ireland, Rovsing is an independent organisation without affiliation to any automotive company. Rovsing provides the following services to the automotive industry: Critical software development services Independent Software Verification and Validation services Advanced solutions for functional testing of hybrid electronic systems IV&V Services Rovsing provides world class IV&V services with a proven track record of return on investment. Rovsing offers a highly flexible approach, allowing the customer to decide at which point in the software development process IV&V activities are included. Rovsing s IV&V activities can add value at each and every stage in the development process; nonetheless, the best results are obtained when IV&V is included from the very beginning. It is our business to be of service to you and allow you the time and peace-of-mind to concentrate on your core business

16 Rovsing A/S Dyregårdsvej 2 DK-2740 Skovlunde Denmark Tel Fax info@rovsing.dk Rovsing Ireland Ltd. Block 1, International Science Centre Rovsing National Ireland Technology Ltd. Park Block Limerick 1, International Science Centre National Ireland Technology Park Limerick Ireland Tel Fax Tel Fax info@rovsing.ie info@rovsing.ie