Dependable Technologies For Critical Systems. Software Verification. 22 nd May Technologies Ltd 2011 Critical Software

Transcription

1 Dependable Technologies For Critical Systems Software Verification 22 nd May 2012

2 Dependable Technologies For Critical Systems Agenda When Things Go Wrong... Certifying Software Safety Critical Systems ISVV Demonstration

3 Dependable Technologies For Critical Systems When Things Go Wrong...

4 Therac 25 Radiation treatment machine for Cancer Was a new unit with greater dependence on software Six massive overdoses between 1985 and 1987 Three deaths directly due to radiation poisoning Root Causes Code not independently reviewed Failure modes not considered Failure codes not explained No system integration testing Complaints not heeded 4

5 Ariane 5 Flight 501 Software code for the Ariane 4 rocket is reused in the Ariane 5 Ariane 5's faster engines trigger a bug in an arithmetic routine inside the rocket's flight computer. The error is in the code that converts a 64-bit floating-point number to a 16-bit signed integer. The faster engines cause the 64-bit numbers to be larger in the Ariane 5 than in the Ariane 4, triggering an overflow condition that results in the flight computer crashing. Result? 5

6 Ariane 5 Flight 501 Fireworks! 6

7 Lauda Air Flight 004 Flight crashed into the jungles of Thailand after the No. 1 thrust reverser inadvertently deployed while the aircraft was at 31,000 feet. All 223 people aboard were killed. 7

8 Blue Screen of Death

9 Blue Screen of Death

10 Dependable Technologies For Critical Systems Certifying Software

11 Why certify software? Today s software: Key component in complex systems Increasing role in reliable systems Impossible to fully test Therefore: We need to have confidence in the software Define and analyse all requirements Develop in a controlled and robust manner Comprehensively test against the requirements The whole process needs to be evidenced 11

12 Why is this important? A steady drive to: Higher levels of automation Ever complex systems Traditional mechanical systems being replaced by software controlled systems Provide confidence to deliver expected functionality safely Meet stakeholder demands (public, environment, business, economic) Long product lifetimes Very expensive if it goes wrong 12

13 The future Further integration of complex systems Higher levels of automation Higher level of functionality Higher feature content 13

14 Key Safety Standards EN ISO Safety Integrity Level (SIL) SIL 1 to SIL4 Derivatives (Domain Specific) EN ISO Railway EN ISO Automotive OLF 070 Oil & Gas DO-178B/C & DO-254 Level A to Level E 14

15 Dependable Technologies For Critical Systems Key Standards: DO-178B / DO-254 Software Considerations in Airborne Systems and Equipment Certification / Design Assurance Guidance For Airborne Electronic Hardware

16 DO-178B and DO-254 DO-178B Developed Many compromises to satisfy goals Able to accommodate different development approaches Objective based Widely adopted DO-178C released January 2012 DO-254 Developed Hardware focussed but very similar to DO-178B for software Covers all electronic hardware 16

17 Why DO-178B & DO-254? Mature and robust Not avionics specific and so easily transferred for use in other industries Proven track record Not a single airline accident caused by software since standards were introduced Used as a benchmark in other industries (e.g. Space) 17

18 DO-178B/DO-254 in Context 18

19 DO-178B Key Features Detailed Planning Five Criticality Levels ( A to E ) Consistency and Determinism Traceability Independence Proven Tools ( Qualification ) 66 Objectives Up to 20 Artefacts 19

20 Criticality Levels Level A 1 X 10-9 Level B 1 X 10-7 Level C 1 X 10-5 Level D >1 X 10-5 Level E NA A. Resulting in Catastrophic Failure B. Resulting in Hazardous/Severe-Major Failure C. Resulting in Major Failure D. Resulting in Minor Failure E. No Effect 20

21 DO-178B Objective based standard DO-178B Assurance level Number of Objectives Independence Cost + Level A % Level B % Level C % Level D % Level E 0 0 0% 21

22 Five Key Plans 1. Planning Process 3. Correctness Process 2. Development Process PSAC SQAP SCMP SDP SVP PSAC: Plan for Software Aspects of Certification SQAP: Software Quality Assurance Plan SCMP: Software Configuration Management Plan SDP: Software Development Plan SVP: Software Verification Plan 22

23 Additional Documents 1. Planning Process 3. Correctness Process 2. Development Process Software Requirements Standard Software Design Standard Software Coding Standard Software Configuration Index (SCI) Software Accomplishment Summary (SAS) Software Traceability Matrix (STM) Requirements, Design, Code and Test Results Tool Qualification Plan CM Records and PR QA & Design Authority audit records Checklists 23

24 V&V Methods 1. Planning Process 3. Correctness Process 2. Development Process Code Inspection Unit Testing Integration Testing Final Integration Testing 24

25 Dependable Technologies For Critical Systems Safety Critical Systems

26 Get it Right! 26

27 V Model 27

28 Verification and Validation Document Plan for each defined life-cycle phase Methods for doing and recording Acceptance criteria Tests For functionality and performance To challenge (stress) the system Validate system to Requirements specifications Intended uses Customer needs 28

29 Verification Activities Reviews Design reviews Walkthroughs Prototyping Traceability matrices Requirements specifications Mitigation of identified hazards Testing Code Scrutiny Module Test Integration Test 29

30 Unit Testing Testing Boundary Testing Robustness Testing Stress Testing Discontinuity Testing Statement Coverage Branch Coverage MC/DC Coverage 30

31 Type of Testing White Box Access to all variables including local variables Grey Box Access to global variables, input/output parameters and functions called. Black Box Only access to global variables and input/output parameters. 31

32 Coverage If( A & B ) { Do this... } Else { Do that... } 32

33 Coverage - Statement If( A & B ) { Do this... } Else { Do that... } Test only needs to cover this branch or the next branch to achieve 100% statement coverage. 1 test case A & B = TRUE OR A & B = FALSE 33

34 Coverage - Branch If( A & B ) { Do this... } Else { Do that... } Test needs to cover this branch and the next branch to achieve 100% branch coverage. Therefore conditions must be TRUE and FALSE. 2 test cases A & B = TRUE A & B = FALSE 34

35 Coverage MC/DC If( A & B ) { Do this... } Else { Do that... } Test needs to verify that both A and B affect the condition. At least 3 test cases A = TRUE, B = TRUE A = FALSE, B = TRUE A = TRUE, B = FALSE 35

36 Validation IMPORTANT Validation is performed before delivery to the customer Validation is performed by personnel that are independent from the design team Examples of Validation Functional and safety tests User acceptance tests Installation and checkout tests 36

37 Sources of Faults Requirements Errors in Conversion Design Incorrect algorithms and interfaces Coding Syntax errors, incorrect signs, endless loops Timing Missed deadlines 37

38 Dependable Technologies For Critical Systems ISVV Demonstration

39 Conclusion System Concept System in Operation Technical Specification Verification ISVV Test Execution and Results Analysis ISVV introduces a small added cost... Software Design Verification Source Code Verification Validation Procedures Implementation Validation Tests Specification...and brings a high added value 39

40 Contacts Russell Jugg Ricardo Silva 40