Application of an Agile Development Process for EN50128/railway conformant

Transcription

1 Application of an Agile Development Process for EN50128/railway conformant Software T. Myklebust SINTEF ICT, Trondheim, Norway T. Stålhane NTNU, Trondheim, Norway N. Lyngby SINTEF ICT, Trondheim, Norway ABSTRACT: The cost of software development is among the major contributors to the total development cost for railway control and signalling systems. It is difficult to make people work faster but it is possible to make them work smarter. One way to achieve that is to introduce agile development methods. Agile methods are gaining increasing popularity, also in safety critical areas such as railway and offshore. Agile methods promise reduced costs and shorter time to market through incremental development, less production of unnecessary documents and more maintainable code. These aspects are also important ingredients in the Shift2Rail joint undertaking focused research and innovation. While the architectural design also in agile development is done up front, detailed design is done incrementally. Based on experiences from other domains, we expect that it will be easier to discover and correct faulty or incomplete system requirements. In addition, agile development will result in simpler code, thus reducing the development and maintenance costs. Only documents that are needed, either for certification or development, are developed. The ones that are developed are used and are thus kept up-to-date. Documents that are not updated are worse than no documentation at all. The challenge is to introduce agile development without compromising safety. Development of signalling systems needs to be compliant with EN This standard impose rigor, but proper adaptation of agile methods can add flexibility and efficiency. In order to evaluate this proposition we use go through all relevant requirements in the standard EN 50128:2011 and mark all requirements as (1) fully met using agile development, (2) possible to meet using agile development and (3) cannot be fulfilled "as is" using agile development. All requirements in category (2) are studied further in order to assign them to category (1) OK, category (2) - some adaptations to the agile method or category (3) changes to the development process. As a final step, we suggest appropriate modifications to the agile development or to the development process. This is the same process that we have used with success for ISO 9001, IEC and IEC The work on IEC resulted in a method named SafeScrum. This model will be reused and modified to fit the current edition of EN However, in order to reap the full benefits of agile development, it is not enough to show conformance to EN The proposed paper will show how agile development can be adapted to fulfill the requirements of EN but also suggested how the current edition of EN should be changed. 1 INTRODUCTION Agile development methods such as Scrum are used in more and more areas of software development, Fitzgerald (2013). For safety critical systems, however, some safety engineers feel that agile development does not fit, the main reason being that these projects require a strict plan which makes later changes costly, Redmill (2014). Still, safety critical projects, like signalling projects, suffer from many of the same problems that mare other software development projects, such as the need to change plans and requirements, being too late and having a solid budget overrun. Our contribution to solving this problem is an assessment of how we can adapt and develop the SafeScrum model Stålhane (2012) to be used by the Railway domain even though the model was developed for the process industry so that it can be used without losing the benefits that we get from these concepts. We have also evaluated the differences in adaption of the SafeScrum model that are necessary for IEC and EN Terminology EN and Scrum First and foremost, Scrum uses incremental development. Since this is an old idea much older than

2 agile development it has a working definition. Incremental development is: A software development technique in which requirements definition, design, implementation, and testing occur in an overlapping, iterative (rather than sequential) manner, resulting in incremental completion of the overall software product. Some of the terms needed for discussion of software development in Scrum are unfortunately not defined outside the agile world. A typical example is a sprint. It can be partly equated with an iteration, but in addition, a sprint needs to contain three activities (1) a sprint planning meeting, (2) a development phase and (3) a sprint retrospective where we sum up lessons learned in the sprint just finished. We will equate the agile term sprint with the standard s term phase. This will, however, create some problems for requirement see a separate discussion related this section later. Another important term is change responsiveness. This term has no generally accepted definition but we will understand it as a project attitude towards changes to the requirements acceptance instead of avoidance. The EN has already taken this into account when they define requirements management as the process of eliciting, documenting, analysing, prioritising and agreeing on requirements and then controlling change and communicating to relevant stakeholders. It is a continuous process throughout a project. Other central terms are not defined either. This goes for the two important terms backlog and daily scrum. For these we have decided to use the definitions supplied in the SolutionsIQ glossary : Backlog: collection of stories and tasks the team will work on at some point in the future. Either the Product Owner has not prioritized them or has assigned them lower priority. Daily scrum: A daily team meeting held to provide a status update to the team members. The semi-real-time status allows participants to know about potential challenges as well as coordinate efforts to resolve difficult and/or time-consuming issues. The meetings are usually time-boxed to 5-15 minutes, and are held standing up to remind people to keep the meeting short and to the point. EN has its own section on terms and definitions in section 3.1 of the standard. None of these definitions prevent us from using an agile approach to software development. We have found it practical to equate component (defined in EN 50128) with functional unit. 1.2 EN The European railway applications standards have been around for more than 10 years and have become so widely accepted that they were adopted as IEC standards, giving them a global applicability. EN is the standard for "Software for railway control and protection systems" and the first edition was issued in Ten years later, it was updated; the 2011 issue shall be "implemented at national level by publication of an identical national standard or by endorsement" by the 25 th of April 2017 at the latest EN (2011)/AC (2014). EN includes detailed requirements related to the organisation, roles and responsibilities. Illustration for a preferred organisational structure of SIL 3 and SIL 4 systems is presented below: Figure 1: Organisation according to EN50128 for SIL 4 Requirements related to documentation are specific in EN and 45 documents are listed in the standard. Personnel competence requirements are strict and concrete involving required key competencies for 10 different roles. Deployment requirements are included to ensure that the software is deployed in the final environment of application. Five named documents have to be produced. 1.3 Configuration management In agile methods there is a need to put more emphasis on configuration management (CM) and change control, including impact analysis, which is currently weak in EN To ensure that all methods and contexts are covered by the next edition of the EN standard a goal-based approach is recommended. In the last two decades there has been an increasing tendency towards a goal-based approach to regulation and standards (requirements for the manufacturers, what they have to do that includes alternative ways of achieving compliance) compared to the earlier prescriptive regulations and standards (requirements that have to be met if a user wishes to claim compliance with the standard). The reasons behind a goal-based approach are rapid technology changes, new development processes and the legal viewpoint. Too restrictive standards may be viewed as a barrier to trade. The CM objective for the development part of CM could be: It should always be possible to restore the system to the state it was at a given time. The resulting requirements for CM could then be shortened to:

3 a) A plan and procedure shall be established to ensure configuration management for unique identification of each items and tools at appropriate times. b) SW changes are managed so that requirements for safety related SW continues to be satisfied. c) guarantee that all necessary operations have been carried out to demonstrate that the required software Systematic Capability has been achieved; d) maintain accurately and with unique identification all configuration items which are necessary to meet the safety integrity requirements of the safety-related system e) apply change-control procedures to prevent unauthorized modifications and register modification requests f) ensure that the following information to permit a subsequent audit: configuration status, release status, the justification for and approval of all modifications, and the details of the modification EN adds a set of requirements on the Scrum software development process. The developers must use the techniques described in the standard s annexes or explain why they use other techniques e.g. why they are just as good as the ones recommended by the standard. Many companies that develop safety critical software have already defined a standard set of development activities based on the required SIL class and their corresponding tools and methods. In this case, the SafeScrum model allows the SafeScrum team to use these tools and methods. 1.4 SafeScrum The adaptation of Scrum to development of safetycritical systems, which we call SafeScrum, is motivated by the need to make it possible to use methods that are flexible with respect to planning, documentation and specification while still being acceptable to EN 50128, as well as making Scrum a practically useful approach for developing safetycritical systems. The rest of this section explains the components and concepts of this combined approach. The SafeScrum model has taken into account that some requirements should be outside an agile approach (separation of concern) and some safety requirements have to be added to the agile methods. Most of the signalling development projects last for several months to several years so we have also included information related to the number of Sprints performed. Separation of concern make possible for an agile approach without overloading it with large amounts of non-agile activities: The software developers develop software in SafeScrum. High level planning, systems design and decisions concerning safety are done outside the SafeScrum process. This is named Separation of Concern, see figure 2 and 3 below. Changes of high-level plans may be fed back from the RAMS validation to the Safe Scrum after each sprint. Figure 2: Waterfall, SafeScrum and Separation of Concern In a Sprint normally the phases 7.4 to 7.7 are included (figure 3). Figure 3: V-model, SafeScrum and separation of concern Add-ons are necessary in the Sprint, this is shown in the figure below.

4 Figure 4: SafeScrum and the Sprint The safety requirements and other requirements are documented as product backlogs. A product backlog lists all functional and safety related system requirements, prioritized by importance. The safety requirements are quite stable (relevant regulations and safety standards are normally stable during the project), while the functional requirements can change considerably over time. Development with a high probability of changes to requirements will favour an agile approach. All risk and safety analyses on the system level are done outside the SafeScrum process, including the analysis needed to decide the SIL level. Software is considered during the initial risk analysis and all later analysis. Just as for testing, safety analysis also improves when it is done iteratively and for small increments. Due to the focus on safety requirements, we use two related product backlogs, one functional product backlog, which is typical for Scrum projects, and one safety product backlog, to handle safety requirements. We will keep track of how each item in the functional product backlog relates to the items in the safety product backlog, i.e. which safety requirements that are affected by which functional requirements. The two backlogs do not necessarily need to be separated by more than different tags within the same requirement tracking tool. The core of the SafeScrum process is the repeated sprints. Each sprint can be a mini waterfall project or a mini V-model, and consists of planning, development, testing, and verification. For the development of safety critical systems, traceability between system/code and backlog items, both functional requirements and safety requirements, is needed. The documentation and maintenance of trace information is introduced as a separate activity in each sprint. In order to be performed in an efficient manner, traceability requires, in practice, the use of a supporting tool. The sprint starts with the selection of the top prioritized items from the product backlog. In the case of SafeScrum, items in the functional product backlog may refer to items in the safety product backlog. The staffing of the development team and the duration of the sprint (14 days is common), together with the estimates of each item decides which items that can be selected for development. The selected items constitute the sprint backlog, which ideally should not be changed during the sprint. The development phase of the sprint is based on developers selecting items from the sprint backlog, and producing code to address the items. A sprint should always produce an increment, which is a piece of the final system, for example executable code. The sprint ends by demonstrating and validating the outcome to assess whether it meets the items in the sprint backlog. Some items may be found to be completed and can be checked out while others may need further refinement in a later sprint and goes back into the backlog. To make Scrum conform to EN 50128, we propose that the final validation in each iteration is done both as a validation of the functional requirements and as a RAMS validation, to address specific safety issues. If appropriate, the independent safety validator may take part in this validation for each sprint. If we discover deviations from the relevant standards or confusions, the assessor should be involved as quickly as possible. Running such an iterative and incremental approach means that the development project can be continuously re-planned based on the most recent experience with the growing product. Figure 5: SafeScrum and several Sprints As the final step, when all the sprints are completed, a final RAMS validation will be done. Given that most of the developed system has been incrementally validated during the sprints, we expect the final RAMS validation to be less extensive than when using other development paradigms. This will also help us to reduce the time and cost needed for certification.

5 The key benefits of this combination of a safetyoriented approach and a process model for agile software development are that the process enables: Continuous communication between customers, the development team and the test team(s). Re-planning based on the most recent understanding of the requirements and the system under development. Mapping of functional and safety requirements. Code-requirements traceability. Coordination of work and responsibilities between the three key roles; the development team, the customer and the assessor. Test-driven development of safety critical systems. All of these points will help us to get a more transparent process and thus better control over the development process. 2 WHY SHOULD THE RAILWAY SIGNALLING INDUSTRY CONSIDER AGILE METHODS? The railway signalling industry has up till now been plan-driven and methodically conservative. Several changes in the environment have, however, affected this: The tempo with which new technology is introduced in the marketplace. This holds both for new constituents and for new on-board and trackside assemblies Increased focus on flexibility and innovation. This is partly a consequence of the first bullet point. There is a growing realization that the plandriven development paradigm is too much focussed on writing and rewriting plans that are not used and on producing documents that are not read. The industry s general focus on lean development and production. Whatever that does not contribute to the product s final value should be removed. There is no reason to believe that the tempo of inventions and innovations will slow down. The railway Joint undertaking S2R (Shift to Rail) and other research initiatives will increase this tempo and those who do not follow will quickly get into trouble. In addition, more and more developers are using agile development. It remains to be seen if these programmers will be interested in working in a development environment based on plan- and document-driven development methods. To quote from our IEC paper Stålhane (2012), the key benefits that comes from this combination of a safetyoriented approach and a process model for agile software development are that the process enables Continuous feedback both to the customer, the development team and the independent test team. Re-planning, based on the most recent understanding of the requirements and the system under development. Mapping of functional and safety requirements. Code-requirements traceability. Coordination of work and responsibilities between the three key roles; the development team, the customer and the assessor. Test-driven development of safety critical systems. All of these points will help us to get a more visible process and thus better control over the development process, which again will help us to deliver on time and within budget. 3 EN AND SCRUM One of the most important concepts when using SafeScrum is the separation of concerns what will be taken care of in the SafeScrum process and what is outside this process. In this way we separate the standard s requirements for software development from the rest of the requirements in the standard. To check possible challenges when adapting SafeScrum for EN compliant software, we have done a detailed study of parts 5 7 of the standard. Section 8 development of application data or algorithms has been left out for now. In addition, we have left out all discussion related to the use of ISO 9001 and the application of change impact analysis since these activities have been described elsewhere see Stålhane (2008) and Stålhane (2014). In addition we have, in an earlier paper, discussed problems related to documentation in agile development see Myklebust (2014). This discussion will not be repeated here. Last but not least, it is important to remain the reader of section in the standard: The life cycle model shall take into account the possibilities of iterations in and between phases. After a thorough walk-through, we are left with the following sections of the standard which will be discussed in more details below: Section 5: Sections organization, Section lifecycle issues Section 6: Section test requirements, Section software verification requirements, Section 6.5 software quality assurance. Section modification and change control

6 Section 7: Section life cycle and documentation for generic software, Section software requirements, Section component design, Section component implementation and testing. The rest of the requirements in section 5-7 are either outside SafeScrum - e.g. architecture - or already taken care of - e.g. safety requirements traceability. Section 5.1.2: subject to assessor s approval, requirements manger, designer and implementer can be the same person. The main point is that the tester must be independent of the implementer. Since these are all roles and not persons, it is not a problem for agile development - it only requires that the tests are written by someone else in the team. See also figure 1. Note that verifier, validator and integrator are new roles which must be added to the Scrum team. Section 5.3.2: life cycle requirements. The standard requires a Software Quality Assurance plan, a Software Verification plan, a Software validation plan and a Software Configuration plan. The standard requires that these plans shall be maintained throughout the development life cycle section This activity has to be inserted into SafeScrum as part of the sprint planning activity. Section 6.1.4: it is important to note that the standard here remove itself from the strict roles defined earlier. In section is says Test performed by other parties such as the Requirements manger, Designer or Implementer, if fully documented and complying with the following requirements (the rest of section 6.1.4) may be accepted by the Verifier. This opens up for developer testing, as is usual in agile development. The important point is to reach an agreement with the Verifier before the development starts. Section gives a detailed set of requirements for a test report. None of these requirements are difficult to implement. The main question is how to do it without slowing down the agile development process unnecessary. In our opinion we need two things (1) a test report template and (2) an adaptation of a test tool printout - e.g. SCRIPT. Requirements d to f can be achieved automatically. Section describes the requirements for software verifications. This implies a new role in SafeScrum. Note that section allows a stepwise development of the verification plan. Section requires us to demonstrate that functional performance and safety requirements are met. This fits well with the SafeScrum separation of concerns. The responsibility described in this section will fit in with the RAMS activity after each sprint. Section 6.5 has been thoroughly discussed in Stålhane (2008). Important issues such as the definition of the life-cycle model and entry- and exitcriteria for each activity are a natural part of agile development. The results presented on documentation in agile development of safety critical software in Myklebust (2014) are also important here. Configuration management (section ) is not a part of an agile development process but is still important. This activity has been discussed with our industrial partners and is easy to add to the SafeScrum process. Section 6.6 change impact analysis is already part of SafeScrum see Stålhane (2014). The only challenge comes from the requirement in section which requires us to go back to an appropriate phase in case of requirements changes. It is, however, up to the project manager to decide what the appropriate phase is. Traceability and configuration management information will help in deciding this. Section gives the requirements for the documents needed for generic software development 46 documents in total - which is describes in annex A1. The majority of these documents (40 out of the 46) can, however, be developed outside SafeScrum even though they will have to be updated after some of the sprints. This is for instance true for the six documents listed under the category Architecture and design. The six documents that need to be considered are all in the categories Component Implementation and Testing (documents 18-20) and Integration (documents 21-23). The only document in the category Component Implementation and Testing that must be considered here is the software source code verification report (document 20) but in our opinion, verification is done as part of RAMS (Reliability, Availability, Maintainability and Safety) and thus outside SafeScrum. Integration documentation (documents 21-23) must be created stepwise as more and more code is integrated after each sprint. This updating process must be part of each sprint retrospective. Section 7.2.1: Software requirements objectives. Section , if taken literally, would prevent the adoption of agile development since it requires that all safety and system requirements should be ready before any coding is done. However, since the standard already allows iterative development, this requirement should be interpreted as before any coding is done in this iteration - in SafeScrum meaning in this sprint. See also requirement in section Section requires the use of the software quality model defined by ISO This is not a problem for development but might be a problem for testing especially testing maintainability usability and portability. It is important for any project, agile or not, to agree with the assessor and the customer how these tests shall be performed. Section and requires that we develop an over-all software test specification. In an agile setting this is done by (1) developing a test

7 specification based on the currently available knowledge and (2) update after each sprint where requirements have been changed, added or removed. Section 7.4 concerns component design. How to relate to this requirement depends on the definition of a component. From the definition given in EN well-defined interfaces and behavior with respect to the software architecture it is reasonable to interpret component as a functional unit - see IEC , section Functional unit: entity of hardware or software, or both, capable of accomplishing a specified purpose. Using this definition, component design is part of system architecture and thus done outside SafeScrum. Section concerns component verification. As mentioned earlier, verifier is a new role that must be added to the Scrum team. In order for SafeScrum to comply with EN 50128, we need the following extensions: Appoint a verifier for the project Software Quality Assurance plan, a Software Verification plan, a Software validation plan and a Software Configuration plan. The plans shall be maintained throughout the development life cycle. It is thus a part of the sprint planning activity in SafeScrum Developer testing is accepted if we reach an agreement with the Verifier on this before the development starts. Demonstration that functional performance and safety requirements are met. The responsibility should lie with the RAMS activity after each sprint We must develop an over-all software test specification by (1) developing a test specification based on the currently available knowledge and (2) update after each sprint where requirements have been changed, added or removed. 4 THREATS TO VALIDITY Related to our conclusion on achieving conformance, we have identified three threats to validity: Have we understood EN 50128? Two of the authors is experienced EN assessor and are involved in certifying safety critical SIL 4 systems for rail signalling and SIL 2 off-shore products. Have we touched all relevant items? By using the SafeScrum process described in section 1.4, we have systematically gone through EN 50128, and we are confident that all relevant items are identified and assessed. Have we understood agile development in general and Scrum in particular? Agile development, including Scrum, is not an exactly defined methodology and there exist a handful of agile methods. Yet they are all based on the few common principles described in section 1.4. We have applied these common and fundamental principles to reduce potential bias from our own interpretations of what agile development is, yet a certain level of subjective interpretation is probably inevitable. Based on the discussion above we claim that our conclusions regarding EN and agile development in general and Scrum in particular will be valid for a wide range of companies and Notified Bodies. 5 CONCLUSION Elements of the agile development process is already used or considered for use in several important industrial domains such as automotive, avionics and industrial automation. When the issues identified in section 3 are settled, it should be straight forward to use SafeScrum and still be EN conformant. It is now important to get one or more companies to try it out in cooperation with the safety assessors to get a reality check of the concepts discussed above. This will allow us to identify possible problems and to make the adjustments necessary for industrial application. With a little flexibility there are, in our opinion, no large obstacles for using agile development for safety-critical software. The main challenges are the EN requirements on organisation, detailed planning, documentation and requirements for proof of conformance. The SafeScrum development process will be an important input to this work. This will lead to a process that is better adapted to handle changes that occur in any software development process and give us an incremental process development, testing and verification that again will lead to more efficient software development. The main differences between IEC and EN are the more stringent requirements in EN related to organization, the validator shall give agreement/disagreement for the software release, documentation and competence requirements. Suggested improvements of EN are more requirements and information regarding modern software development methods. This is also in accordance with preliminary work performed by the current IEC maintenance committee.

8 6 REFERENCES EN Railway applications Communication, signalling and processing systems Software for railway control and protection systems. Ed.2 Fitzgerald et al Scaling Agile Methods to Regulated Environments: An Industry Case Study Myklebust, T., Stålhane T., Hanssen G. K., Wien T. and Haugset B Scrum, documentation and the IEC :2010 software standard. PSAM 12 Hawaii Redmill F. Agile methods for developing safety-related software?. SCSC Newsletter September 2014 Stålhane, T. and Hanssen, G.K The application of ISO 9001 to agile software development. Stålhane, T., Myklebust, T., and Hanssen, G.K The application of Safe Scrum to IEC certifiable software. ESREL 2012, Helsinki, Finland Stålhane, T. Hanssen, G. K. Myklebust T. and Haugset B Agile change impact analysis of safety critical software. SafeComp_Sassur