Ingegneria del Software II academic year: Course Web-site: [

Transcription

1 Course: Ingegneria del Software II academic year: Course Web-site: [ Dependability and Software Qualities Lecturer: Henry Muccini and Vittorio Cortellessa Computer Science Department University of L'Aquila -Italy [ [

2 Copyright Notice» The material in these slides may be freely reproduced and distributed, partially or totally, as far as an explicit reference or acknowledge to the material author is preserved. Henry Muccini 2

3 Acknowledgment» This material comes from: - Ian Sommerville, Software Engineering 7th edition, chapters 20, 24, and 3 - Ian Sommerville, Software Engineering 6th edition, chapter 16 - Henry Muccini, ICS 122 course on Software Specification and Quality Engineering, year 2002, 3

4 Topics covered» Dependability» Dependable processes» Software Qualities and Dependability - Reliability - Safety - Security» Other qualities 4

5 Dependability The extent to which a critical system is trusted by its users 5

6 Software dependability» In general, software customers expect all software to be dependable. However, for non-critical applications, they may be willing to accept some system failures.» Some applications, however, have very high dependability requirements and special software engineering techniques may be used to achieve this. 6

7 The concept of dependability» For critical systems, it is usually the case that the most important system property is the dependability of the system» The dependability of a system reflects the user s degree of trust in that system. It reflects the extent of the user s confidence that it will operate as users expect and that it will not fail in normal use» Usefulness and trustworthiness are not the same thing. A system does not have to be trusted to be useful 7

8 Dependability achievement» Fault avoidance - The system is developed in such a way that human error is avoided and thus system faults are minimised. - The development process is organised so that faults in the system are detected and repaired before delivery to the customer.» Fault detection - Verification and validation techniques are used to discover and remove faults in a system before it is deployed. 8

9 Costs of increasing dependability Cost Dependability 9 Low Medium High Very high Ultrahigh

10 Dependability costs» Dependability costs tend to increase exponentially as increasing levels of dependability are required» There are two reasons for this - The use of more expensive development techniques and hardware that are required to achieve the higher levels of dependability - The increased testing and system validation that is required to convince the system client that the required levels of dependability have been achieved 10

11 Dependable processes» To ensure a minimal number of software faults, it is important to have a well-defined, repeatable software process.» A well-defined repeatable process is one that does not depend entirely on individual skills; rather can be enacted by different people.» For fault detection, it is clear that the process activities should include significant effort devoted to verification and validation. 11

12 Validation activities» Requirements inspections.» Requirements management.» Model checking.» Design and code inspection.» Static analysis.» Test planning and management.» Configuration management is also essential. 12

13 Dimensions of dependability Dependability Availability Reliability Safety Security The ability of the system to deliver services when requested The ability of the system to deliver services as specified? The ability of the system to operate without catastrophic failure The ability of the system to protect itelf against accidental or deliverate intrusion 13

14 Quality Factors Classification Functional: what the software does integrity reliability survivability usability Performance: how well it does it correctness efficiency safety interoperability Change: modifying the software maintainability expandability flexibility portability reusability Management: planning, controlling, testing, installation verifiability manageability 14

15 Exercise 1) B2C Web Application: - We want to analyze a Web system in which users can view and buy products - The payment may be done on-line - Different categories of products may be found 2) Nuclear Power Plant: - A Sensor component stores information/parameters on a Db (e.g., temperature, pressure) - A Control component checks parameters from the Db - When parameters value are unexpected, the Control stops the system 15» What are the quality attributes of this system?

16 Topics covered [Sommerville 7, ch 24]» Reliability validation» Safety assurance» Security assessment» Safety and dependability cases 16

17 Validation of critical systems» The verification and validation costs for critical systems involves additional validation processes and analysis than for non-critical systems. Such costs are worthy when: - The costs and consequences of failure are high so it is cheaper to find and remove faults than to pay for system failure; - You may have to make a formal case to customers or to a regulator that the system meets its dependability requirements. This dependability case may require specific V & V activities to be carried out. 17

18 Reliability [ICS 122 course]» Reliability deals with the rate of failures in the software that render it unusable - reliability is a statistical property: theprobability that software will operate without failure over a specified period of time, where failures include > software returns incorrect results > results are not accurate enough > response time is too slow > software stops without recovering» Fitness for use regarding reliability 18 time between failures is acceptable

19 Reliability (cont.) [ICS 122 course]» Rareamong commercial software groups» More applied in medical systems, military groups, on board weapons systems - and other software that controls the operation of large and complex machinery» Reliability measurements: - Mean time to failure (MTTF) - Mean time between failures (MTBF) 19

20 Reliability validation» Reliability validation involves exercising the program to assess whether or not it has reached the required level of reliability.» This cannot normally be included as part of a normal defect testing process because data for defect testing is (usually) atypical of actual usage data.» Reliability measurement therefore requires a specially designed data set that replicates the pattern of inputs to be processed by the system. 20

21 The reliability measurement process Identify operational profiles Prepare test data set Apply tests to system Compute observed reliability» Establish the operational profile for the system.» Construct test data reflecting the operational profile.» Test the system and observe the number of failures and the times of these failures. 21» Compute the reliability after a statistically significant number of failures have been observed.

22 Exercise» Apply this reliability measurement process to the selected case study 22

23 Statistical testing» Testing software for reliability rather than fault detection.» Measuring the number of errors allows the reliability of the software to be predicted. Note that, for statistical reasons, more errors than are allowed for in the reliability specification must be induced.» An acceptable level of reliability should be specified and the software tested and amended until that level of reliability is reached. 23

24 Reliability measurement problems» Operational profile uncertainty - The operational profile may not be an accurate reflection of the real use of the system.» High costs of test data generation - Costs can be very high if the test data for the system cannot be generated automatically.» Statistical uncertainty - You need a statistically significant number of failures to compute the reliability but highly reliable systems will rarely fail. 24

25 Operational profiles» An operational profile is a set of test data whose frequency matches the actual frequency of these inputs from normal usage of the system (the same frequency use expected in normal usage). - A close match with actual usage is necessary otherwise the measured reliability will not be reflected in the actual usage of the system.» It can be generated from real data collected from an existing system or (more often) depends on assumptions made about the pattern of usage of a system. (e.g., telecom systems) 25

26 Operational profile generation» Should be generated automatically whenever possible.» Automatic profile generation is difficult for interactive systems and for new and innovative systems» Operational profile is also difficult when dealing with users with different skills» Maybe straightforward for normal inputs but it is difficult to predict unlikely inputs and to create test data for them.» Referece: Musa 1998, Software Reliability Engineering: More reliable Software, Faster Development and Testing 26

27 An operational profile Number of inputs Input classes... 27

28 Reliability prediction» It is really important to stop testing as soon as possible and not overtest the system» Areliability growth model is a mathematical model of the system reliability change as it is tested and faults are removed.» It is used as a means of reliability prediction by extrapolating from current data - Simplifies test planning and customer negotiations. - You can predict when testing will be completed and demonstrate to customers whether or not the reliability growth will ever be achieved.» Prediction depends on the use of statistical testing to measure the reliability of a system version. 28

29 Observed reliability growth» The equal-step growth model is simple but it does not normally reflect reality.» Reliability does not necessarily increase with change as the change can introduce new faults.» The rate of reliability growth tends to slow down with time as frequently occurring faults are discovered and removed from the software.» A random-growth model where reliability changes fluctuate may be a more accurate reflection of real changes to reliability. 29

30 Growth model selection» Many different reliability growth models have been proposed.» There is no universally applicable growth model.» Reliability should be measured and observed data should be fitted to several models.» The best-fit model can then be used for reliability prediction (Kan 2003, model) 30

31 Equal-step reliability growth [Jelinski and Moranda, 1972] Reliability (ROCOF) Rate of Occurrence of software failures t1 t2 t3 t4 t5 Time 31

32 Random-step reliability growth [Littlewood and Verral, 1973] Reliability (ROCOF) Rate of Occurrence of software failures Note different reliability improvements Fault repair ad ds ne w fault and decreases reliability (increases ROC OF) t1 t2 t3 t4 t5 Time 32

33 Reliability prediction [Kan, 2003] Reliability = Measur ed reliability Fitted reliability model curv e Required reliability Estima ted time of reliability achie vement Time 33

34 Safety [ICS 122 course]» Safety deals with the absence of unsafe software conditions» Safety def: theexecution of software within a system context without resulting in unacceptable risks and the absence of danger by using the software» Safety critical systems systems in which computers are capable of causing death or injury (e.g., Trmcs), or catastrophic events to the environment» Fitness for use regarding safety software can be trusted to avoid hazardous behavior 34

35 An example [ Fatal Defect book, Ivars Peterson, 1995, page37]» Therac-25 - Incident > radiation-treatment machine for cancer therapy patients > A combination of rare keystrokes > Overdose of radiation to patients > 6 killed or injured - Problem > Poor software engineering practices along w/ building a machine that relies on the software for safe operation. 35

36 An example» Sizewell B nuclear power plant - 10,000 lines of code used to shut down the nuclear plant in caseof an increase in temperature. - When inspected it failed about 1/2 of them. - Failure was blamed on the test not the software. - Declared safe but lowered the chance of an accident from 10,000 to 1,

37 Safety assurance» Safety assurance and reliability measurement are quite different: - Within the limits of measurement error, you know whether or not a required level of reliability has been achieved; - However, quantitative measurement of safety is impossible. Safety assurance is concerned with establishing a confidence level in the system. 37

38 Safety confidence» Confidence in the safety of a system can vary from very low to very high.» Confidence is developed through: - Past experience with the company developing the software; - The use of dependable processes and process activities geared to safety; - Extensive V & V including both static and dynamic validation techniques. 38

39 Safety arguments» Safety arguments are intended to show that the system cannot reach in unsafe state.» These are weaker than correctness arguments which must show that the system code conforms to its specification.» They are generally based on proof by contradiction - Assume that an unsafe state can be reached; - Show that this is contradicted by the program code. 39» A graphical model of the safety argument may be developed.

40 Construction of a safety argument» Establish the safe exit conditions for a component or a program.» Starting from the END of the code, work backwards until you have identified all paths that lead to the exit of the code.» Assume that the exit condition is false.» Show that, for each path leading to the exit that the assignments made in that path contradict the assumption of an unsafe exit from the component. 40

41 Safety related process activities» Creation of a hazard logging and monitoring system.» Appointment of project safety engineers.» Extensive use of safety reviews.» Creation of a safety certification system.» Detailed configuration management (see Chapter 29). 41

42 Hazard analysis» Hazard analysis involves identifying hazards and their root causes.» There should be clear traceability from identified hazards through their analysis to the actions taken during the process to ensure that these hazards have been covered.» A hazard log may be used to track hazards throughout the process. 42

43 Hazard log entry Hazard Log. Page 4: Printed System: Insulin Pump System Safety Engineer: James Brown File: InsulinPump/Safety/HazardLog Log version: 1/3 Identified Hazard Insulin overdose delivered to patient Identified by Jane Williams Criticality class 1 Identified risk High Fault tree identified YES Date Location Hazard Log, Page 5 Fault tree creators Jane Williams and Bill Smith Fault tree checked YES Date Checker James Brown System safety design requirements 1. The system shall include self-testing software that will test the sensor system, the clock and the insulin delivery system. 2. The self-checking software shall be executed once per minute 3. In the event of the self-checking software discovering a fault in any of the system components, an audible warning shall be issued and the pump display should indicate the name of the component where the fault has been discovered. The delivery of insulin should be suspended. 4. The system shall incorporate an override system that allows the system user to modify the computed dose of insulin that is to be delivered by the system. 5. The amount of override should be limited to be no greater than a pre-set value that is set when the system is configured by medical staff. 43

44 Run-time safety checking» During program execution, safety checks can be incorporated as assertions to check that the program is executing within a safe operating envelope.» Assertions can be included as comments (or using an assert statement in some languages). Code can be generated automatically to check these assertions. 44

45 Security assessment» Security assessment has something in common with safety assessment.» It is intended to demonstrate that the system cannot enter some state (an unsafe or an insecure state) rather than to demonstrate that the system can do something.» However, there are differences - Safety problems are accidental; security problems are deliberate; 45 - Safety problems are mostly related to the application domain. Securityproblems are more generic -many systems suffer from the same problems.

46 Security validation» Experience-based validation - The system is reviewed and analysed against the types of attack that are known to the validation team.» Tool-based validation - Various security tools such as password checkers are used to analyse the system in operation.» Tiger teams - A team is established whose goal is to breach the security of the system by simulating attacks on the system.» Formal verification 46 - The system is verified against a formal security specification.

47 Key points» Reliability measurement relies on exercising the system using an operational profile -a simulated input set which matches the actual usage of the system.» Reliability growth modelling is concerned with modelling how the reliability of a software system improves as it is tested and faults are removed.» Safety arguments or proofs are a way of demonstrating that a hazardous condition can never occur. 47

48 Key points» It is important to have a dependable process for safetycritical systems development. The process should include hazard identification and monitoring activities.» Security validation may involve experience-based analysis, tool-based analysis or the use of tiger teams to attack the system.» Safety cases collect together the evidence that a system is safe. 48

49 Software Qualities (from the ICS122 course) 49

50 Integrity» Integrity deals with security against either overt or covert access to program or data - unauthorized access, which may be accidental or deliberate» Integrity guarantee that: - the information cannot be modified and that - the information arrives to the recipient as it has been sent» Also implemented through one-way hash functions 50 hash(t) hash(d), " T D

51 Security» Access Control» Authentication» Confidentiality» Data Integrity» Traffic Flow Integrity» Assured Usage 51

52 Survivability» Survivability deals with the continuity of reliable software execution in the presence of system failures - possibly with degraded functionality - robustness» Survivability is the ability to complete a mission successfully in the face of hostile fire (in weapon systems.)» Survivability is the ability of a network computing system to provide essential services in the presence of attacks and failures, and recover full services in a timely manner (in network systems)» Fitness for use regarding survivability user can still use essential functions even though some part of system is down 52

53 Usability» Usability deals with how easy the software is to use (by all users) - initial effort required to learn > enhanced or degraded by readability of documentation - recurring effort to use > enhanced or degraded by naturalness of user interface - general functionality of the software - usability is an extremely subjective property» Fitness for use regarding usability software is easier to use than not to use for software team, software is understandable 53

54 Correctness» Correctness deals with the extent to which the software design and implementation conform to stated requirements - correctness concerns include > all specified functions are implemented (functional correctness) > design is documented according to standard (document correctness) > performance of software is within constraints (performance correctness) - a requirements specification must be available for which it is possible to determine whether the software is consistent» Fitness for use regarding correctness what was produced is what was specified 54

55 Efficiency» Efficiency deals with resources needed to provide required functionality - resources include > processor > memory > disk space > communication» Fitness for use regarding efficiency resources required for software execution are affordable 55

56 Interoperability» Interoperability deals with how easy it is to couple software with other systems or applications - coupling involves integration - interoperability is enhanced by defining standard interfaces in application domains» Fitness for use regarding interoperability software produces or uses results that comply with industry standards and/or provides an open interface that can be used by others Middlewaresand APIs 56

57 Maintainability» Maintainability deals with the ease of changing and modifying the system - Corrective maintenance: removal of residual faults, or bugs, in software after delivery (~20%) > note: this does not include adaptive or perfective maintenance» Maintainability becomes important when systems are required to be used for long time - Year 2000 problem» Fitness for use regarding maintainability software is productive during maintenance, defect detection and correction is appropriately handled by issuance of a new release 57 for software team, software is easy to modify

58 Expandability» Expandability deals with the perfective aspects of software maintenance - increasing the software s functionality or performance to meet new user needs - Perfective maintenance: changing software to improve qualities (~50%)» Fitness of use regarding expandability software was built to be open ended 58 for software team, software is easy to modify and add new capabilities

59 Flexibility» Flexibility deals with the adaptive aspects of software maintenance» The ability of the product option to incorporate future additions, augmentations or expansions without requiring you to purchase a new version of the same product - adaptive maintenance: modifying software to operate in different environments (~30%)» Fitness of use regarding flexibility 59 software is readily adaptable to a wide range of environments without resorting to expandability-type changes

60 Portability» Portability deals transporting the software to execute on a different platform - platform could be > hardware: processor > software platform: operating system, compiler» Software needs to able to adapt to new hardware configurations.» Fitness of use regarding portability software may be used on several different platforms without resorting to flexibility-type changes 60

61 Reusability» Reusability deals with use of portions of software for other applications - perhaps with minor modification» COTS and CBSE» Regression Testing - Ariane 5 failure» Defect reduction and integration problems» Fitness of use regarding reusability 61 software consists of a large library of building blocks that can be used somewhat independently

62 Verifiability» Verifiability deals with how easy it is to determine if it satisfies the desired attributes - Verifiability can be performed by automated testing and analysis procedures - Verifiability can be improved by monitoring the desired properties - It needs an adequate definition of desired attributes» Fitness of use regarding verifiability simple to certify that the software is correct 62

63 Manageability» Manageability deals with the administrative aspects of software modification - manageability includes > tools to support changes: both planning and control > media control: installation» Fitness of use regarding manageability support environment is adequate and easy to use 63

64 Related Operational Qualities» Suitability: presence and appropriateness of functionality for specified task» Accuracy: provision of right precision and effects» Compliance: adherence to application-related standards or conventions» Fault Tolerance: ability to maintain specified level of performance in case of faulty software» Recoverability: ability to re-establish level of performance or recover data in case of failure 64

65 Related Operational Qualities» Understandability: user s effort in recognizing the logical concept and applicability» Learnability: user s effort in learning software s application (operational control, input, output)» Operability: user s effort for operational control 65

66 Related Maintenance Qualities» Understandability: developer s effort in recognizing the logical concepts in the code» Analyzability: effort required for diagnosis of deficiencies or cause of failure» Stability: risk of unexpected effect of modification» Testability: effort needed to test software to validate modifications (or correctness) 66

67 Related Maintenance Qualities» Installability: effort needed to install software in specified environment» Conformance: adherence to standards or conventions related to portability» Replaceability: opportunity and effort of using software in place of other system in the environment 67

68 To Conclude» There are more than 50 different quality attributes that could be analyzed» but we are not interested in doing that» There is the need to PRIORITIZE quality attributes, in order to be able to create a rank: - 5. Mandatory - 4. Very important - 3. Rather important - 2. Not important Does not Matter