The Costs, Benefits, and Risks Associated With Pattern -Based and Modular Safety Case Development

Transcription

1 The Costs, Benefits, and Risks Associated With Pattern -Based and Modular Safety Case Development Dr Tim Kelly, MA PhD; University of York; York Simon Bates, MEng; University of York; York K e y w o r d s : Safety cases, Safety Case Patterns, Modular Safety Cases, Goal Structuring Notation, GSN Abstract Safety Case Patterns, and their implementation using the Goal Structuring Notation (GSN), were first introduced in (ref. 1). They provide a mechanism for capturing and reusing common arguments (such as ALARP arguments) within safety cases. Modular safety case development is concerned with the decomposition of complex safety cases into well-defined separate but interlinked modules of argument and evidence. Extensions to GSN to support modular safety case development were first proposed in (ref. 2). Both modular and pattern-oriented safety case development approaches have been proposed as mechanisms for improving the quality of, and reducing the risks associated with, safety case development for complex systems. To investigate these risks and benefits, together with gaining a better understanding of associated costs, MoD and QinetiQ (as part of the ARP project Improved Availability and Reduced Lifecycle Costs for Military Avionics ) commissioned York to undertake a study. This study investigated current experience and opinion regarding modularity and patterns through a series of interviews. Safety case practitioners from MoD and industry, representing a variety of roles in safety case development (developers, consultants, managers, and assessors) were involved in the study. The key findings of the study are presented. Consequent recommendations for further work to improve safety case development are also made. Introduction This study was requested by the MoD and funded by QinetiQ as part of the ARP Funded Research Project Improved Availability and Reduced Lifecycle Costs for Military Avionics. The scope of the study was to investigate the costs, benefits, and risks associated with modular and pattern-oriented safety case development compared with traditional development. Modular and pattern-oriented safety case development techniques have been proposed as mechanisms for improving the quality of, and reducing the risks associated with, safety case development for complex systems. This study investigated current opinion regarding modularity and patterns in safety case development through a series of interviews with the aim of establishing a better understanding of the following: Differences in costs between establishing a traditional and a modular safety case for a new system Differences in cost when maintaining a traditional and a modular safety case during the life of a system Risks associated with modular and pattern-oriented safety case development Benefits of adopting a pattern-oriented approach to safety case development Benefits of modular re-certification over re-certification of a traditional safety case following change Page 1 of 14

2 Background to the Study Safety Case Patterns, and their implementation using the Goal Structuring Notation (GSN), were first introduced as a concept in (ref. 1). They have been recognised as a mechanism for capturing experience learnt during safety case development. This experience is documented by extracting commonly occurring and mature parts of a safety case, i.e. parts that can be sufficiently generalised so that they can be adapted and instantiated in other safety cases. It has been commonly recognised that this more systematic approach to reuse reduces risk. Modular safety development is a technique associated with the move towards incremental certification, i.e. certification as separate argument modules are produced and changed. This technique has been proposed and is being developed to improve the production of safety cases for complex systems. Extensions to GSN to support modular safety case development were first proposed in (ref. 2). Whilst modular safety case development is frequently discussed and the potential benefits associated with it declared to be great, there is as yet limited practical experience of using the approach. Study Method It was decided to conduct the study through a series of interviews. To prepare for these interviews a review of existing published safety case development experience was conducted. Reports such as (refs. 3, 4, 5, 6, 7) were reviewed and from these several (both perceived and experienced) benefits and risks were extracted. The core of the interview was structured around asking interviewees for their response (i.e. level of agreement / disagreement) to a series of statements associated with these benefits and risks. In addition, time was also given in the interviews for open questions to capture interviewees experiences that lay outside of the scope of the pre-defined benefit and risk statements. Within the timescales of the study, it was not possible to conducted a full quantitative assessment of the comparative costs of modular and pattern-oriented safety case development and maintenance (compared to traditional safety case development.) Instead, interviewees were asked to use their experience (citing specifics where possible) to respond to a series of qualitative statements regarding safety case management costs. Interviewees were selected to reflect a cross-section of the different roles commonly found within the safety case domain. The roles identified were generalised from (ref. 9) into the following four categories: developers, maintainers, managers, and assessors. The interviews were conducted in February and March In all, 29 interviews were conducted. The interviews were generally well received and the opinion was expressed that the interview conducted a thorough investigation into modular and pattern-based safety case development techniques. The findings of the interviews are reported and analysed in the next section. S t u d y R e s u l t s The following three sections present an overview of the study findings regarding pattern-oriented safety case development, modular safety case development, and the costs associated with adopting these approaches in safety case management practice. Page 2 of 14

3 Pattern - Oriented Safety Case Development Results Figure 1 provides the full list of the benefit and risk statements concerning pattern-oriented safety case development that were presented to interviewees. This section will discuss the main findings from these questions. P a t t e r n B e n e f i t s 1. Safety Case Patterns capture safety case development expertise and improve maturity 2. Safety Case Patterns help improve argument completeness 3. Safety Case Patterns provide the inspiration or a starting point for new safety argument developments 4. Safety Case Patterns help those with little safety case experience 5. Safety Case Patterns help in planning and scoping safety cases 6. Safety Case Patterns help speed up safety case development 7. Safety Case Patterns provide a benchmark when reviewing a safety case P a t t e r n R i s k s 1. Patterns increase costs due to the effort required to extract and document commonly occurring argument structures 2. Maintaining an up-to-date record of usage and evolution of patterns increases costs 3. Dogmatic application of patterns in development and maintenance increases costs 4. Inappropriate selection of patterns can lead to incorrect safety arguments 5. Patterns can stifle creativity and limit identification of more optimal solutions 6. Patterns based arguments can help developers receive undeserved credibility 7. Patterns only help in structuring higher-level arguments often leaving you to struggle with lower-level details F i g u r e 1 P a t t e r n- O r i e n t e d S a f e t y C a s e D e v e l o p m e n t B e n e f i t s a n d R i s k s Pattern - O r i e n t e d S a f e t y C a s e D e v e l o p m e n t B e n e f i t s Interviewees most strongly associated Pattern Benefit 3 (Patterns provide the inspiration or a starting point for new safety argument developments) with pattern-oriented safety case development. 94% of interviewees either agreed (41%) or strongly agreed (53%) with this statement. Interviewees generally expressed the opinion that patterns provide a good starting point for safety argument construction. The following statements summarise the interviewees open responses: Patterns provide a starting point for discussions of argument approach Patterns not only provide a means of creating an argument, but they also provide a means of discussing different argument approaches. It was recognised that even if a pattern was not eventually used in a safety case it was a useful technique for improving the maturity of safety case development. Another benefit recognised was that patterns provide a good way of quickly developing the more commonly occurring arguments so that development effort can be focussed on the more novel aspects of the safety case Page 3 of 14

4 Patterns make it easier to produce an argument for a systems where similar systems have already been certified - Patterns were felt to provide an experienced view of what has previously worked in particular environments, regulatory contexts, and for particular systems. Patterns help produce outline arguments for bids A number of interviewees responded by saying that patterns had provided them with a way of more quickly preparing indicative outline safety arguments for contract bids. Coupled with this was the experience that when contracts were eventually won patterns had helped reduced the time and effort, and therefore costs, associated with building the safety case. Interviewees also raised the following concerns regarding Pattern Benefit 3: Significant up-front effort is required to define good Safety Case Patterns A frequently expressed opinion was that patterns need to be used with thought and skill. Another frequently expressed opinion was patterns provide you with a starting point, but they do not do the work for you. Even though patterns improve confidence and understanding of a safety case approach, effort and skill still need to be used to ensure that an instantiation or adaptation of a pattern does not result in the loss of the good starting point benefit. There is currently limited (or no) access to libraries of Safety Case Patterns from which safety case patterns can be selected This observation indicates an immaturity of the approach. Whilst most interviewees recognised the benefits that Safety Case Patterns could give them, a number expressed the opinion that these benefits were not yet completely realisable. This was also reflected by the various responses prompted by Pattern Risk 7 (Patterns only help in structuring higher-level arguments often leaving you to struggle with lower-level details). The opinion frequently expressed was that patterns should be useful at all development levels, but currently they are not owing to the scarcity of currently available patterns. Of all the pattern benefits proposed, interviewees least strongly associated Pattern Benefit 5 (Patterns help in planning and scoping safety cases) with pattern-oriented safety case development. However, even though it was the least supported benefit statement, 62% of interviewees still either agreed or strongly agreed with the statement. Therefore, it can be concluded that the majority of people would consider using patterns to help planning and scoping safety case developments. Further support for this benefit was given in the interviewees open responses, summarised in the following statements: Patterns provide a starting point for discussing the argument approach Patterns provide a pre-prepared argument basis through which the scope and completeness of the safety case can be improved, and through raised awareness provide a better means of planning the safety case approach. Patterns help in setting and managing expectations for safety case developers A number of interviewees commented that patterns can be used to provide safety case developers with a skeleton outline of the required approach. Within this skeleton developers can develop and gather evidence. Page 4 of 14

5 Interviewees also raised the following concerns regarding Pattern Benefit 5: Limited libraries from which complete safety cases can be generated Without comprehensive pattern libraries, patterns cannot be used to help plan and scope a complete safety case. Instead, they can only be used to plan parts of the safety case. Therefore, to improve the techniques usefulness in safety case planning and scoping, it would be beneficial to improve the completeness of existing pattern libraries. Pattern - Oriented Safety Case Development Risks Interviewees most strongly associated Pattern Risk 4 (Inappropriate selection of patterns can lead to incorrect safety arguments) with pattern-oriented safety case development. 94% of interviewees either agreed (34%) or strongly agreed (60%) with this statement. Many thoughts as to how and why patterns can be used inappropriately were given in the interviewees open responses. These are summarised as follows: Patterns cannot just be advocated, people need to be educated and each pattern application justified It was recognised that inexperienced safety case developers may unthinkingly adopt a Safety Case Pattern without fully understanding its suitability and appropriateness for a specific project. To combat this, most interviewees recognised that either patterns should only be used by experienced practitioners, or that developers must be educated about patterns and their application. Interviewees also suggested that another way of protecting against Pattern Risk 4 was requiring justification of the use of a pattern to be documented. Limited libraries from which complete safety cases can be produced Interviewees recognised that, with small numbers of patterns to choose from, there was potential to force a inappropriate pattern as a solution to a safety argument development problem. Therefore, another way of protecting against Pattern Risk 4 would be to improve the number of Safety Case Patterns available in existing pattern libraries. Thus potentially reducing the likelihood that any single safety case pattern would be forced to fit. There were also a number of reasons given by interviewees that should mean that Pattern Risk 4 should not occur. These are summarised as follows: Patterns aid discussions and provide a common basis about which people can agree/disagree This benefit, stated by a number of the interviewees, indicates that patterns facilitate discussions about safety cases, and help in reaching a common shared understanding of the safety case approach required on any given project. Therefore, another means of mitigating Pattern Risk 4 is not to mandate usage of patterns, but instead to use the pattern as a communication medium to raise understanding of the required safety case approach. The interviewees considered Pattern Risk 3 (Dogmatic application of patterns in development and maintenance increases costs) to be the biggest project risk associated with pattern-oriented safety case development. 80% of interviewees either agreed or strongly agreed with this risk statement. Reasons as to why Pattern Risk 3 was considered to present the biggest project risk were revealed through the responses given to the open questions. These have been summarised as follows: Page 5 of 14

6 Patterns should only be part of an experienced developers toolkit The premise behind this statement was that novices would tend not to question the content of the expert view conveyed by the pattern. As a result, it was felt that patterns could lead to dogmatism if inexperienced or unskilled developers used them. Therefore, it could be concluded that patterns are not a tool for novices. However in contradiction to this statement, 79% of interviewees either agreed or strongly agreed with Pattern Benefit 4 (Patterns help those with little safety case experience including in training). Therefore, a better conclusion would be that patterns present the biggest project risk when developers or managers that are unskilled or unfamiliar with patterns use them without question. Patterns could encourage people to stop thinking and take shortcuts This risk was commonly identified by interviewees and can be associated with the toolkit statement above. However, it is identified here as separate concern because regardless of experience or skill people could, for a number of reasons, look to take shortcuts, and patterns could be misused in this way. Therefore, to protect against this occurring the use of patterns needs to be managed and justified as well being advocated. A number of counter-arguments to the risks associated with the dogmatic application of patterns were also voiced. These are summarised as follows: Patterns provide a starting point for thinking about the safety case Anything that makes you think should protect against dogma. However, it was recognised that even though patterns provide a medium to help you think, they also provide a medium for taking shortcuts and not thinking. Therefore, in order to protect against dogmatism the use of patterns as a vehicle for thought and discussion needs to be encouraged. Patterns capture learnt experience and improve the maturity of safety case development This benefit could be interpreted as one of the reasons why dogmatism can occur i.e. because patterns present an expert view the underlying principles are not questioned. However, if patterns are used appropriately they present a medium through which a safety case can be greatly improved or, as most interviewees claimed, they help ensure a minimum quality for safety arguments. Interviewees least associated Pattern Risk 1 (Patterns increase costs due to the effort required to extract and document commonly occurring argument structures) with pattern-oriented safety case development. The majority of interviewees responded by saying that whilst it was true that institutionalising patterns could present large up-front costs, these costs would be more than outweighed by the long-term benefits. Interviewees least associated Pattern Risk 7 (Patterns only help in structuring higher-level arguments often leaving you to struggle with lower-level details) with pattern-oriented safety case development. 62% of interviewees either disagreed or strongly disagreed with this benefit. A common response to this question was that, whilst current pattern libraries were incomplete, and that most patterns they were aware of catered only for the high level parts of the safety case, there was no reason that patterns could not be produced to help with the production of lower level arguments. Therefore, it can be concluded that this risk would be mitigated if existing pattern libraries were extended to incorporate patterns that aid argument development at all levels of the safety case. Page 6 of 14

7 Open Responses Regarding Pattern- Oriented Safety Case Development From the open questions the most interesting responses made in favour of pattern-oriented safety case development were as follows: Safety Case Patterns help ensure a minimum standard for safety cases There will be initial costs involved in setting up patterns. However, these costs can be spread across several projects. Each time a pattern is reused there is a cost benefit. Safety Case Patterns would benefit from being supported by a software development tool. It was hoped that such a tool could be packaged with existing pattern libraries and a means of creating and categorising your own patterns. Safety Case Patterns can help reviewers highlight gaps in the safety cases. Patterns can help in modular safety case construction, e.g. through providing exemplars of arguments concerning interactions and independence. From the open question responses the most interesting responses made against pattern-oriented safety case development were as follows: If a pattern is improved, then you may be asked to reapply the pattern to previous instantiations, especially if improved during a project. Use of patterns could result in a dangerous managerial view that the task of producing a safety case can be deskilled. Modular Safety Case Development Results Figure 2 provides the full list of benefit and risk statements associated with modular safety case development that were presented to interviewees. This section will discuss the main findings from these questions. Modular Safety Case Development Benefits Interviewees most strongly associated Module Benefits 5 (Modular development improves toplevel planning of the safety case in the large ) and 7 (A modular structure helps manage organisational and/or contractual boundaries) with modular safety case development. Interestingly, these two benefits relate to modular safety case structure (Module Benefit 5) and safety case module interfaces (Module Benefit 7), indicating that whilst there are lots of improvements gained from modularity the major benefits to be gained are improved safety case structure and more formal associations between parts of a safety case. There were a number of open responses given by interviewees that provided further support for Module Benefit 5. These are summarised as follows: Modularity results in the ability to better plan for change and identify the potential for reuse Due to the coarse-grained nature of a modular safety case a number of interviewees responded that this should allow them to better discuss how to improve the change and reuse potential of a safety case. The results of these discussions could then be used to a) improve safety case structure to better support change and reuse, and b) to feedback into the drawing up of processes so that handling of change and reuse is improved. Page 7 of 14

8 M o d u l a r B e n e f i t s 1. Modular safety case development supports work division and work sharing 2. An explicit modular safety case structure helps in tracking and dependencies between safety arguments 3. An explicit modular safety case structure reduces the rediscovery and review effort (e.g. the effort associated with locating and tracking inter-safety case dependencies) 4. An explicit modular safety case structure limits the effects of change 5. Modular safety case development improves top-level planning of the safety case in the large 6. An explicit modular safety case structure promotes reuse within, and across, argument application 7. An explicit modular safety case structure helps manage organisational and/or contractual safety case boundaries Modular Risks 1. Modular development unnecessarily complicates safety arguments 2. Safety case modules and their structure provide false confidence regarding change resilience 3. A modular safety case increases the effort of tracking inter-module dependencies 4. Cross module safety issues can often be poorly handled 5. Poor alignment of the modular safety case structure with the system s organisation results in the loss of the benefits of modularity 6. Safety case modules facilitate inappropriate and dangerous reuse of safety arguments 7. Poor handling of module composition can result in an invalid safety case F i g u r e 2 Modular Safety Case Development Benefits and Risks Divide and conquer should mean that management can more easily focus skills, assign responsibilities, and schedule completion deadlines - Most interviewees commented that modularity supported the divide and conquer of the safety case. A few interviewees went on to say that they would expect that, because of divide and conquer activity, more traditional management styles could be adopted when planning the safety case, which would result in a safety case that was more in line with a project and, therefore, more manageable than a monolithic safety case. Modular safety case development provides a common framework for the multitude of different actors involved to discuss and communicate issues to management This was identified as a benefit to top-level planning by most interviewees. They claimed that because you had better visibility of the overall safety case it was both easier to spot gaps or problems, and discuss possible arguments earlier in the safety case development. In spite of the strong support for this benefit there were some risks and problems raised in discussion of Module Benefit 5. These have been summarised as follows: Increased management effort in getting buy-in from all parties? It could be argued, based on other open responses received, that getting buy-in can be just as problematic in traditional safety case development and that modular safety case production goes some way to reducing Page 8 of 14

9 the effort of obtaining buy-in. With a modular safety case structure you may be better able to plan the safety case, and possibly involve all parties (such as suppliers and assessors) in the planning process. As was identified by a number of interviewees, this involvement should result in better all-round confidence and support of the safety case approach and result in the production of a more compelling safety case. Developers and managers might not see important safety case issues up-front, leading to problems later? Based on other open responses received it could be argued that, given that a modular safety case structure improves discussion and visibility of the safety case, this risk would be reduced in comparison to traditional safety case development. There were a number of open question responses that offered support for Module Benefit 7: Divide and conquer should mean that management can more easily focus skills, assign responsibilities, and schedule completion deadlines Formal responsibilities can be assigned to modular arguments and, as a knock-on effect, more formal boundaries established. These boundaries can then, be used to manage the separation between organisations and/or contracts. Most interviewees recognised this as providing the significant benefit of being able to more clearly assign and manage ownership of arguments. Therefore, safety case module interfaces were recognised as not only providing a medium to separate concerns, but also a medium through which inter-organisational relationships can be managed Well-defined modular safety case interfaces allow organisations to see how they should contribute to the overall safety case The ability to see what an organisation is expected to contribute should make managing relationships in safety case development easier. There were also a number of open responses received that offered counter-arguments to Module Benefit 7: Completeness of safety case interfaces could have serious consequences if insufficient If interfaces were insufficiently recorded cross-module issues could be incorrectly or poorly handled, and incompatibilities between safety case modules (e.g. mismatched assumptions) not revealed. There was felt to be little guidance available to help with this issue and, therefore, further investigation would be beneficial to establish the best way to identify and capture safety case interfaces to ensure completeness Safety case interfaces are the biggest risk. Significant management effort should be expected in their identification and enforcement Concerns were raised regarding the definition of safety case interfaces, capturing and conveying interface information, and enforcing the interface. There was consensus that modular safety case development techniques lacked guidance as to how these three issues should be addressed, and as such it was an area most interviewees expressed as requiring further investigation. Interviewees least associated Module Benefit 4 (A modular safety case structure limits the effects of change) with modular safety case development. However, 62% of interviewees still either agreed or strongly agreed with this statement. Most interviewees commented, whilst responding to this statement, that a modular safety case would not actually limit the effects of change. Page 9 of 14

10 Instead, it would allow easier identification of the impact of change on the safety case resulting in change effort being focussed on the change impact, and hence reducing costs by enabling project teams to more quickly react with a proportionate response. Modular Safety Case Development Risks Interviewees most strongly associated Module Risk 7 (Poor handling of module composition can result in an invalid safety case) with modular safety case development. 90% either agreed or strongly agreed with this statement. As such it should be recognised as a significant concern associated with modular safety case development. Various reasons were given by the interviewees for this concern. These are summarised as follows: Safety case interfaces can provide a false sense of security Poor handling of safety case composition was recognised by interviewees as a significant risk. The incomplete specification of safety case module interfaces will lead to problems when composing safety cases modules. During safety case interface development subtle issues of context and technical requirements might be missed It was felt further guidance is necessary concerning how to capture relevant safety case interface information. Interviewees strongly disagreed with Module Risk 1 (Modular development unnecessarily complicates safety arguments). 79% of interviewees either disagreed or strongly disagreed with this risk. Whilst the majority of interviewees recognised that this was indeed a risk, the frequent response was that, by the very nature of modularity, the argument contained within a particular safety case module would be as easy, if not easier, to develop than an argument for a traditional style safety case. It was also commented that modular safety case development might actually help novices, owing to the fact that modularity aids handling the overall complexity of an argument. Open Responses Regarding Modular Safety Case Development The following additional benefits of modular safety case development were voiced by interviewees: Modularity supports evolutionary development meaning that achieving incremental certification should be easier to achieve. It should be possible to identify arguments that are similar across projects and coordinate development of common arguments, which can be reused in, and adapted for, multiple projects. The following additional risk associated with modular safety case development was voiced by interviewees: Modular safety case development must be supported by corresponding modular safety case acceptance and certification processes. Such processes are not yet being practiced. Page 10 of 14

11 Anticipated Initial Costs and Cost- S a v i n g s A s s o c i a t e d w i t h P a t t e r n - Oriented and Modular Safety Case Development Interviewees were asked to make experience-based judgements (where possible and estimates where not) as to the costs involved with institutionalising pattern-oriented and modular approaches to safety case development as part of their organisations everyday safety case development practice. Interviewees were also asked to make experience-based judgements (where possible and estimates where not) as to the long-term cost savings they would expect to gain from introducing these practices. 50% of interviewees expected that institutionalising pattern use as part of the safety case development process would require an increase in up-front project safety case costs of 0-10%. It should be noted however, that 38% of interviewees opted not to express an opinion. However, even those who opted out from formally responding commented that there would be inevitable up-front costs associated with training, tool support, and establishing patterns for use as part of the developer s toolkit. The majority (66%) of interviewees expected pattern-use to result in a 0-25% cost reduction (of current safety case development costs). The cost benefit (long-term) was therefore considered to outweigh the (up-front) costs. 34% of interviewees elected not the respond to this question. However, it was generally expected (even by those who refused to be drawn on a figure) that there would be long-term benefits from using patterns including more efficient argument production and improved quality of safety arguments. The general opinion of the interviewees was that pattern-oriented safety case development ultimately presents both costs and benefits. A way of improving the perceived cost to benefit ratio was highlighted by one interviewee who suggested that the favoured approach would be to get organisational buy-in to pattern-based safety case development such that the costs of institutionalising the approach could be spread across a number of projects. This would mean that each project would contribute a small amount to the total up-front costs, but could expect to get greater benefit in return. 79% of interviewees expected that institutionalising a modular approach to safety case development process would require an increase in up-front project safety case costs of between 0-25%. It was generally recognised that modular safety case development would result in increased initial costs associated with establishing a modular structure and identifying interfaces. Also, on the first time through the costs were likely to be high because tool support would need to be acquired and mistakes would almost certainly be made. 65% of interviewees expect to get a long-term cost benefit (i.e. reduction) of between 11-50% (of current safety case development costs) from using a modular approach to safety case development. Many of the interviewees commented that for the first certification of a modular safety case the costs are likely to be high, however it was expected that these costs would be dramatically reduced when it comes to re-certification. Combining these responses it can concluded that the overall expectation of the interviewees was that initial costs associated with introducing modular safety case development will be outweighed by the long-term benefits. Page 11 of 14

12 Recommendations Arising The following recommendations arise from the findings of the study: It was generally recognised that the completeness of existing Safety Case Pattern libraries is an area requiring further development. Coupled to this there is, in the opinion of a few interviewees, a need for a central and universally accessible repository for safety argument patterns. Therefore, it recommended that two projects, one to extract and document patterns and the other to set up a centralised pattern repository, should be conducted. The majority of interviewees expressed a concern that safety case interfaces present the biggest challenge and risk to modular safety case development and that this is, therefore, an area requiring further work. Several key areas for further work have been identified: Extracting information for interfaces further guidance is required regarding extracting the relevant information to be captured in safety case module interface descriptions. Communicating interfaces further investigation is required regarding how to capture and document interfaces in an intuitive and manageable form. Capturing assurance further work is required regarding how to capture information concerning the strength of arguments contained and required by safety case modules as part of the safety case module interfaces. Further investigation is desirable regarding how interfaces can/should be used to improve legal and contractual relationships between organisations It was generally accepted that with both techniques there are up-front costs, but that these will be more than outweighed by the benefits. However, there was some concern that estimates of the costs and effort associated with modular safety case development were often too low. This is an area recommended for further study to establish a realistic picture of the costs and effort associated with introducing modularity to safety case developments. Tool support was a concern for a number of interviewees and affects both techniques. It is recommended that an investigation be conducted first to gather opinion on what features such a tool should have, and then later investment be made in the development of a tool. It was recognised that a standard reference for GSN was required to provide vital project support and improve the commercial longevity of the technique, especially to improve the adoption of Safety Case Patterns and modular safety cases. Therefore, it is recommended that this is an area requiring further investment. It was recognised that modular development could result in several parts of the argument being sent for review / acceptance at the same time. However, whilst parallel development promised the benefits of increased productivity, these benefits could be counteracted by the serial nature of safety case review and acceptance processes. Therefore, it is necessary to conduct further studies to ascertain whether or not it would be beneficial to alter review and acceptance processes to better support modular safety case development. Page 12 of 14

13 Conclusions From these results it can be concluded that the majority of the opinions polled estimated that the long-term benefits of both pattern-based and modular safety case development more than outweigh the (initial) costs. This is therefore a positive indication that the improvements presented by the techniques are cost effective and worth the initial investment in time and effort. However, it was recognised that neither technique was without flaw. A number of possible areas of further investigation have been identified that, if pursued, would help improve the maturity of the techniques and reduce the project and technical risks. The recommendations summarised above indicate, that patterns as a technique are understood, but that problems stem from the size of, and access to, existing pattern libraries. The recommendations also indicate that the adequacy of current approaches to capture and document modular safety case interfaces is a significant concern. Overall, the study has shown that it is widely believed that investment in both pattern-based and modular safety case development will reap a long-term benefit. However, in order for these techniques to survive the test of time underlying support needs to be improved. The support required is threefold 1) organisational support for the techniques 2) software tool support, and 3) additional training and guidance. A c k n o w l e d g e m e n t s This study was requested by the MoD and funded by QinetiQ as part of the ARP Funded Research Project Improved Availability and Reduced Lifecycle Costs for Military Avionics. The authors would like to acknowledge the financial support given by QinetiQ for the work reported in this paper. Re ferences 1. T P Kelly, J A McDermid, Safety Case Construction and Reuse using Patterns in Proceedings of 16th International Conference on Computer Safety, Reliability and Security (SAFECOMP'97), September 1997, Springer-Verlag 2. T P Kelly, Concepts and Principles of Compositional Safety Cases Research Report commissioned by QinetiQ, COMSA/2001/1/1 (available from 3. European Organisation For The Safety Of Air Navigation, The EUR RVSM Pre- Implementation Safety Case, Version 2.0, 14th August 2001, EUROCONTROL 4. C H Pygott, Compositional Safety Cases For COTS Assessment, Technical Report QinetiQ/KI/TIM/TR021996, March C H Pygott, Assessment of Operating Systems For Safety Related Applications, Technical Report QinteiQ/KI/TIM/TR , June P Chinneck, D J Pumfrey, and J A McDermid, The HEAT/ACT Preliminary Safety Case: A Case Study In the Use Of Goal Structuring Notation, In Proceedings of the 9th Australian Workshop Safety Critical Programmable Systems (SCS'04), Brisbane, Conferences in Research and Practice in Information Technology, Vol. 38, Australian Computer Society, P Chinneck, D J Pumfrey, and T P Kelly, Turning Up The HEAT On Safety Case Construction, in Proceedings of 12th Safety Critical Systems Symposium (SSS'04), February 2004, Springer Verlag Page 13 of 14

14 8. T P Kelly, Managing Complex Safety Cases in Proceedings of the 11th Safety Critical Systems Symposium (SSS'03), February 2003, Springer Verlag 9. H W Lawson, An Assessment Methodology for Safety-critical Systems in Proceedings of the 12th Annual CSR Worshop / 1st ENCRESS Conference, 1995, Springer-Verlag Biographies Dr Tim Kelly, Department of Computer Science, University of York, York, YO10 5DD, UK. Phone: , Fax: tim.kelly@cs.york.ac.uk Dr Tim Kelly is a Lecturer in software and safety engineering within the Department of Computer Science at the University of York. He is also Deputy Director of the Rolls-Royce Systems and Software Engineering University Technology Centre (UTC) at York. His expertise lies predominantly in the areas of safety case development and management. His early research focussed upon safety argument presentation, maintenance, and reuse using the Goal Structuring Notation (GSN). Tim has provided extensive consultative and facilitative support in the production of acceptable safety cases for companies from the medical, aerospace, railways and power generation sectors. He has published over 50 papers on safety case development and software safety in international journals and conferences and has been an invited speaker on software safety issues. Mr Simon Bates, Department of Computer Science, University of York, York, YO10 5DD, UK. Simon Bates was a Research Associate within the BAE SYSTEMS funded Dependable Computing Systems Centre (DCSC) at the University of York from October 2002 to December Simon worked on the study discussed in this paper from January 2005 to March Simon has since left the University to train as a patent attorney. Page 14 of 14