[control] [data] [process] A practical approach to using Agile in an FDA regulated environment environment Jim Gunning Director, Q-CSV Johnson & Johnson [strategy] [partners] [testing] [validation]
Agenda What is Agile? Regulatory expectations for software development Agile vs. Waterfall Agile in Practice Benefits and Challenges Confidential. For J&J internal use only. 2
What is Agile? Agile is a group of software development methods in which requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. Agile Implementation track used in J&J is based on the Agile SCRUM methodology Allows for iterative and incremental software development where requirements and the solution itself evolve based on feedback from cross-functional teams It promotes adaptive planning and minimizes the cost of change Confidential. For J&J internal use only. 3
Agile values and principles Agile promotes adaptive planning, evolutionary development, early delivery, continuous improvement, and encourages rapid and flexible response to change. Individuals and interactions over Working software over Processes and tools Comprehensive documentation Customer collaboration over Responding to change over Contract negotiation Following a plan Confidential. For J&J internal use only. 4
Regulatory Expectations for software development Regulatory agencies do not prohibit or encourage the use of any specific software development methodology, but they do indicate some expected characteristics of the selected software lifecycle and development: Established (controlled) processes for producing high-quality software (control over design and development processes) Evidence of design control, or how the requirements set has evolved during the development of the product, and require risk management to be integrated into the design process Software VERIFICATION and VALIDATION conducted throughout the software development lifecycle Documentation (evidence) to demonstrate compliance to regulations and to facilitate the investigation of software problems Confidential. For J&J internal use only. 5
Reconciling the Scrum Approach with Regulations Scrum Expectations Working software over comprehensive documentation Responding to change over following a plan Individuals and interactions over processes and tools Regulatory Expectations Established (controlled) processes for producing high-quality software (control over design and development processes) Evidence of design control, or how the requirements set has evolved during the development of the product, and require risk management to be integrated into the design process Software VERIFICATION and VALIDATION conducted throughout the software development lifecycle Documentation (evidence) to demonstrate compliance to regulations and to facilitate the investigation of software problems Regulatory agencies do not prohibit or encourage the use of any specific software development methodology, but they do indicate some expected characteristics of the selected software lifecycle and development. Confidential. For J&J internal use only. 6
Comparing Waterfall to Agile Waterfall inherently wants to lock down requirements in the beginning of development and isn t designed to handle change well In reality change during the computer system development is unavoidable and in some cases beneficial Change can come from a variety of drivers and can impact development Agile does not attempt to lock down requirements in the beginning and assumes requirements and design will evolve and change as more is known during the course of sprinting (Cone of Uncertainty) Agile breaks projects down into user stories which are little chunks of user functionality End users and business stakeholders get to see and experience the system early in the development process Quality improves because testing starts in the sprints Risk is reduced because the risk process is incorporated early Confidential. For J&J internal use only. 7
The Role of Documentation in Waterfall Produce documentation that is valuable to both the development team and regulatory stakeholders. Requirements /Design Build Test/Validate In a linear model such as waterfall, in which process activities are executed by different people over different times, documentation provides a means for those people to communicate over time. In the flow of activities shown above, each activity would produce documentation to be handed off to the person who needs it in the subsequent activity. Users(Business) produce requirements documentation to be handed off to developers (IT) to design, who then produce design documentation to be handed off to designers to do develop testing QA approves various documents and testing based on an understanding of they system and design as described in requirement and specification documents Confidential. For J&J internal use only. 8
The Role of Documentation in Agile In a parallel model such as Agile, the need for documentation changes significantly. A single team consists of people playing multiple roles and working on multiple activities at the same time so there is less need and less value in producing written documentation as the main mechanism of communication between those people and activities. Documentation still maintains its importance in communicating the results of a team s activities and there are two important audiences to consider: External stakeholders, such as customers, regulators, and auditors, who need information about what was done in order to assess the product and the processes used to create it. Considering regulators and auditors, the team must consider, When we are done, will we have sufficient evidence to demonstrate that our quality system processes were followed correctly? The second audience is a development team that will take the results of the team s activities to build upon in some future work Confidential. For J&J internal use only. 9
Agile documentation Who are you communicating with? Once we consider documentation a way of communicating it s worth considering how best to communicate with each audience. Internally (Scrum team) The Scrum team should choose the best way to communicate. Often the most efficient way to communicate will be face to face a key agile principle. Where teams need more precise or long lasting communication they will create documents that are useful to them, such as the product backlog. Stakeholders outside the team In waterfall projects, much of the communication outside of the team is provided by status updates by project managers. In agile, the frequent availability of working software makes this the new method of communication. Working software communicates progress and achievement of stakeholder s requirements clearly and effectively. Project managers still provide status updates and other documentation will describe more precisely what the software does but this documentation only supports working software. It describes what has been built as compared to waterfall documentation s focus on what will be built. Regulatory partners (eg. QCSV) As with other stakeholders outside the team the primary method of communication in agile is through presentation and testing of working software. As this is a key relationship however, communication should occur as early and frequently as possible. It will involve face to face communication as well as documents such as the Agile Compliance Plan, Product backlog, testing scripts/records etc. Confidential. For J&J internal use only. 10
Agile documentation This slide lists some of the key agile documentation. Where a project is a linked to a previous waterfall release it may be appropriate to use some of that documentation. Existing legacy life of the system waterfall documentation can be updated when new scope has an impact on existing requirements. Also mapping of waterfall documentation and agile documentation will help when new scope has an impact on existing requirements. Product backlog This is where the requirements for the project are kept. Functional requirements, regulatory and security requirements etc. are all broken down here into items called user stories. These express an aspect of what what the product should do from a single perspective (eg. User/admin etc.) Agile compliance plan This document outlines how the Team will ensure that all SDLC deliverables and activities throughout the Agile project fulfil compliance requirements. Testing scripts/records These records are an essential part of producing high quality code and also form a vital part of the traceability chain. They should demonstrate what has been tested, what the acceptance criteria are, who did the testing etc. And should allow tracability back to the requirement (and reason for that requirement) the code meets. Confidential. For J&J internal use only. 11
Agile Traceability When considering traceability in agile projects, it s worth considering its two roles; as useful documentation and for regulatory compliance. To achieve these roles without impacting time spent developing working software, you may need to consider the toolset you use to achieve traceability Traceability as useful documents As traceability records are effectively documents, we should try to apply the agile principle of Working software over comprehensive documentation. This means that creating the traceability records should take less less time and effort to do that it saves later in the project (by assisting in areas such as testing/bug reporting etc.) Traceability for regulatory compliance Traceability is necessary to provide a trail of changes and decisions that can be followed for software that is being developed under regulatory compliance. Traceability toolset The first approach is to consider the tooling used during development. As far as possible, development, testing and application lifecycle management tools should interoperate to create traceability automatically e.g. storing unit tests with developed code, allowing tests to be linked to user stories in the product backlog. This helps ensure that the traceability records help to create value (in the form of working software) rather than eating into value by taking up more time than they save. This may mean that you need to evaluate the toolset you have used in your project under waterfall to ensure that it can support an effective agile approach. If it does not, special attention must be paid to how to transfer any traceability records from one tool to another. Useful documents Regulatory compliance Toolset Confidential. For J&J internal use only. 12
Agile Testing In waterfall, the testing phase often falls towards the end of the project. From one point of view this is understandable this is the first time that enough pieces of the application are complete enough to enable them to be tested together. Unfortunately we also know that the end of the project is when the cost of any code changes is at its highest. A key agile principle is that we Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. The key words in this principle are working software. By delivering working software every sprint (no matter how small that functionality may be) we have something that can be tested much earlier and more often. In agile development, testing is a part of every sprint, not something that waits until the end of a project. Working in agile goes further than just testing at the end of each sprint though. Once a team s tasks have been prioritized and assigned correctly at the start of sprint, how can teams ensure that what they are producing is working software? The solution is test-driven development. Developers build automated testing before they build the code that it tests. The tests are stored alongside the code to help developers know when code is working or done. These tests also serve as documentation that helps to provide a part of the traceability path, showing what the code is supposed to do and the expected behaviour for parts of the software. WATERFALL TESTING WORKING SOFTWARE TEST-DRIVEN DEVELOPMENT Confidential. For J&J internal use only. 13
Testing for regulated applications Testing of applications being developed for regulatory environments (such as GxP) requires attention to some particular elements. Firstly it should be noted that early engagement with QCSV will help to ensure that regulatory requirements will be captured and clearly available to the agile team as soon as possible in the project. Contact should be maintained with these groups to ensure that these requirements are kept up to date throughout the project. The general changes to where testing takes place in agile compared to waterfall remain. Agile testing takes place in every sprint whereas testing to meet regulatory requirements is often late in a project. There are currently two approaches to testing for GxP applications. Option 1 Qualified QA test environment Alternatively, teams can set up a validated environment before beginning sprint 1 in conjunction with QCSV and the regulatory requirements. 1. Set up and qualify a development and a QA environment 2. Set up documentation for completion and approvals with QCSV 3. Map acceptance criteria to test scripts and conditions. This approach enables QCSV to carry out their reviews of testing/code within each sprint. Qualified test environment Option 2 Hardening Sprint Teams can carry out a final hardening sprint before release where the application is tested against the regulatory regulatory requirements that are set with QCSV. During a hardening sprint applicable documentation updates (such as sprint backlog and design/configuration design/configuration documentation for GxP) are completed and approved, pre & post approval of test scripts are obtained, code is moved to and formal testing performed in a qualified environment. Hardening sprint Confidential. For J&J internal use only. 14
Risk-Based Validation Testing REQUIREMENTS BUSINESS PROCESSES /REQUIREMENTS Potential hazards? Root causes of failure? BUSINESS PROCESSES SUPPLIER ASSESSMENT GxP COMPLIANCE RISK RISK SCORE + RISK PRIORITY H M L = BUSINESS PROCESS / REQUIREMENT RISK MITIGATION CONTROLS & RISK-BASED VALIDATION TESTING Recommended testing types by business process / requirement level risk Cross-functional team conducts risk assessment 1 2 3 4 Confidential. For J&J internal use only. 15
GxP Compliance Risk Use the GxP compliance risk and the business process/requirement risk to determine risk-based validation testing Compliance Analysis HIGH DIRECT IMPACT MEDIUM INDIRECT IMPACT LOW REMOTE IMPACT Risk-Based Approach to Validation SOP BP / REQ T RISK BP / REQ T RISK BP / REQ T RISK LOW MORE LEVEL OF TESTING LESS Confidential. For J&J internal use only. 16
QUALITY RISK MANAGEMENT SOP for Risk-Based Approach to Validation Confidential. For J&J internal use only. 17
Regression Test Strategy All test scripts manually executed and Q-CSV approved assessed for regression Regression Test Bank created and approved Scripts are automated if possible Multiple regressions can be planned depending on code changes Confidential. For J&J internal use only. 18
Key Compliance Deliverables Compliance documentation managed in DocSpace and ALM Some documents created/approved once and others incrementally with sprints Key Documents Compliance Analysis Master Compliance Plan Compliance Plan Master Test Plan Release Sprint Test Protocol (covers all testing except Dev and UAT) IQ Protocol IQ Scripts Sprint Backlogs (User Stories) User Story Risk Assessments Functional Specifications (Embedded in user story acceptance criteria) Technical Design Specifications Test Scripts, Defects, Trace Matrix (ALM) Data Conversion/Migration Strategy Data Conversion/Migration Protocol Regression Strategy Integration Strategy (Interfaces) IQ Summary Report Release Sprint Summary Report UAT Protocol UAT Report Compliance Summary Report Master Compliance Summary Report Build Sprints Compliance Documentation Confidential. For J&J internal use only. 19
Benefits Closer (earlier) collaboration between CSV and Development Testing started much earlier in the beginning of development End users seeing system earlier and identifying issues early Better opportunity for course corrections within sprints Doing something cutting edge that has brought out the best from the team in terms of creativity and problem solving Confidential. For J&J internal use only. 20
Challenges Balancing the theoretical with the practical Environment constraints development for building and QA for formal testing Unable to perform UAT in sprints because of SOP readiness Determining and balancing correct level of CSV involvement in Dev Sprints Keeping Independent Quality Limited data in Release Sprints because of the speed of sprints as compared to ETL process Difficult to get incremental approval of compliance documentation because of speed and timing related to sprinting Compliance workarounds related to non-validated tools (JIRA) Challenges related to off-shore model Confidential. For J&J internal use only. 21
Questions Confidential. For J&J internal use only. 22