Achieving Regulatory and Industry Standards Compliance with the Scaled Agile Framework

Transcription

1 Achieving Regulatory and Industry Standards Compliance with the Scaled Agile Framework March 8, 2017 Dr. Steve Mayner SAFe Senior Program Consultant Trainer Scaled Agile, Inc. V Scaled Agile, Inc. All Rights Reserved Scaled Agile, Inc. All Rights Reserved. Harry Koehnemann Director of Technology, 321 Gang 1.1

2 Agenda 4Regulatory requirements meet Agile development 4Meeting regulatory requirements with Lean-Agile development Build the solution and compliance incrementally Organize for value and compliance Build quality and compliance in Apply continuous verification and validation 4Summary 1.2

3 Regulatory requirements meet Agile development 1.3

4 Traditional development challenges regulatory assessment 4Often results in late verification, validation, and opportunities for compliance assessment 4Can create a large bow wave of testing and compliance activity resulting in missed deadlines and windows for schedule compliance activities 4Despite best efforts, compliance with large batches and late feedback slows flow and results in worse outcomes and lower quality 4No way to leverage compliance knowledge to improve flow 1.4

5 Regulatory and compliance meet Agile Regulatory and compliance 4Quality, safety, security, efficacy 4Specifications 4Verification and validation 4Objective evidence 4Inspections, audits, sign-off 4Quality management systems 4Metrics defects, requirements coverage, code coverage, traceability Agile Manifesto Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan Welcome changing requirements. 1.5

6 Verification, validation, and compliance Verification is the confirmation through objective evidence that the specified requirements have been fulfilled. Verification demonstrates that the design and implementation correctly and completely embody the requirements. We built the solution right. Validation is the confirmation via objective evidence that the system performs its intended function. The intended functions and how well the system performs those functions are determined by the customer. We built the right solution. Compliance is an organization s adherence to relevant laws, specifications and guidelines. Compliance typically requires documented, objective evidence of solution V&V through tests, inspections, analysis, or demonstrations. And we have evidence that we have done so. 1.6

7 Our QMS addresses compliance and other concerns 4Safety and efficacy concerns come from many sources Regulatory Customer 4QMS defines policies, processes, and procedures to meet relevant regulations Statutory Industry best practices 1.7

8 Two types of compliance requirements On the Solution Ensure product complies Sensitive information must be encrypted during transmission over networks that are easily accessed by malicious individuals PCI Credit Card Standards Stopping from 20 miles per hour in a distance [ ] that is not greater than the distance specified in the table in paragraph (d) CFR Title (automotive) On the Quality Management System Ensure process complies A documented software requirements specification (SRS) provides a baseline for both validation and verification. -- US FDA 21 CFR 820 The purpose of the Quality Assurance Process is to provide independent and objective assurance that work products and processes comply with predefined provisions and plans and that non-conformances are resolved and further prevented -- ISO 15504, ASPICE 1.8

9 However, Waterfall legacy has embedded itself in our QMS Waterfall development creates many problems Underestimated dependencies Massive growth in complexity Late delivery Little visibility into process Early commitment to specification that was insufficient Phase gates aren t helping reduce risk No way to improve systematically Late discovery of issues Any notion that we are mandated to apply a single-pass, waterfall model to software development is an industry myth, one which has likely been perpetuated by our own waterfall past ( we have always done it this way ) and our existing quality management system, and not because the regulations make us do it. -- Dean Leffingwell 1.9

10 Knowledge for people building the world's most important systems SAFe is a freely revealed knowledge base of integrated, proven patterns for enterprise Lean-Agile development

11 The Big Picture view 1.11

12 SAFe addresses both types of compliance requirements Product is specified, designed, built and tested in accordance with regulations Objective Evidence Processes and procedures are performed in accordance with policy based on regulations Sign-off Approval Certification Validation 1.12

13 Build the solution and compliance incrementally Organize for value and compliance Build quality and compliance in Apply continuous verification and validation Meeting regulatory requirements with Lean-Agile development 1.13

14 Build the solution and compliance incrementally 1.14

15 Specify, build, and comply incrementally A P C D One PDCA cycle Documents Documents More Documents Unverified System Certified Solution A P A P A P A P A P C D C D C D C D C D 1.15

16 Iterate within the product lifecycle Conceive Build Verify and Validate Refine business need Postulate solution Create roadmap to realize Use cadence to execute and adapt based on learning Continually refine solution, intent, and roadmap Regularly align teams and integrate solution Converge on optimal solution End users confirm decisions Validate in operational environment Exploration feasibility Product committed Solution ready Limited distribution General distribution Market analysis A C P D A C P D A C P D Previous system specs Domain experts 1.16

17 Provide objective evidence at every increment Solution progress Delivery progress Relentless improvement Objectives Vision Roadmap Quality Compliance Manufacturability Customer feedback Regulatory feedback Validated learning Improvement stories PI Demo PI PI PI 1.17

18 Organize for value and compliance 1.18

19 Take a systems view: understand the full Value Stream A fundamental thinking construct in Lean Trigger Lead time $ REPEAT 4Each Value Stream is the sequence of steps used to deliver value to the Customer 4It includes the whole sequence, from concept or customer order to delivery of value and/or receipt of cash 4It contains the people who do the work, as well as the flow of information and materials 1.19

20 Value streams cut across organizational silos Business Product Mgt/ Hardware Software Quality Testing V&V Security Sys. Engineering 1.20

21 Cross-functional Agile Release Trains deliver value Business Product Mgt/ Hardware Software Quality Testing V&V Security Sys. Engineering AGILE RELEASE TRAIN (ART) V&V 1.21

22 Integrate regulatory, customer, and compliance concerns Kanban and Backlog Align and focus on work priorities Roadmap Know solution s future plan PI Demo Frequent feedback on fullyintegrated system PI Planning Synchronizes on near term plan Inspect & Adapt Continuously improve together Solution Intent Shared understanding of system s goals, specifications, and constraints PI 1.22

23 Build quality and compliance in 1.23

24 Build quality practices into process as part of flow 4Continuous integration 4Test-First 4Refactoring 4Pair-work 4Collective ownership 4 Exploratory early iterations 4 Model Based Systems Engineering (MBSE) 4 Set-Based Design 4 Frequent integration 4 Design verification 4 Reviews, audits, sign-offs 4 V&V, IV&V 4 Quality assurance activity 4 Regulatory oversight 4 Coverage, completeness 1.24

25 MBSE facilitates emergent specifications 4Evolve specifications each increment 4Ensure alignment across all solution builders 4Generate specifications and compliance documents from models to ensure single source of truth MBSE Fuel Performance Ignition ECM Subsystem Spec Feature Spec Component Spec Subsystem Spec v2 Feature Spec v2 Component Spec v2 1.25

26 Accelerate feedback through automated compliance testing Give teams automated scripts instead of checklists 4Automate tests in the same iteration as the functionality 4Include tests for safety, security, performance, quality, etc. Quality Safety V&V Security ü Functional test û Security test ü Performance test QA test û Compliance test ü ü ü ü ü Functional test Security test Performance test QA test Compliance test 4Invest in automated testing infrastructure to improve flow Progress Done 4Actively maintain test data under version control Building functionality Functional test automation Compliance test automation 1.26

27 Continually reduce risk for meeting compliance objectives 4Break compliance activities into smaller batches 4Regular cadence make compliance predictable Reduce last sign-off activity from a large, extended event to a quick, boring, non-event. 4Get visibility, transparency into assessment sooner 4Fast feedback continuously improves practices V&V 1.27

28 Include compliance concerns in continuous improvement Solution Demo Asses results of current compliance work and automated tests Demonstrate compliance status by assessing or generating latest information and documents Quantitative Measurement Show trends towards meeting compliance solution and dev system % code coverage, % requirements coverage, peer reviews status % compliance concerns covered in automated tests Retrospective and Problem Solving Workshop Are we sufficiently addressing compliance goals? Are policies or procedures inhibiting development? If so, can we find alternatives to meeting compliance goals 1.28

29 Apply continuous verification and validation 1.29

30 Solution Intent captures requirements and compliance info Trace regulation to place it s implemented and tests that demonstrate it Requirements Model Customer, regulatory Functional and use cases NFRs safety, security Traceability Solution Model Analysis Architecture/interfaces Domain Model Business process Business rules Realization Code, hardware Documentation Manufacturing, installation Test Model Test plans, test cases, execution results Compliance documents and information 1.30

31 SAFe requirements model supports continuous verification Backlog items Solution specification and constraints Enable functionality Build functionality Constrains functionality Enabler Exploration Architecture Infrastructure Evaluate & feedback Feature tested by Feature acceptance test realized by * Story acceptance test Story tested by Unit test Strive for automation Nonfunctional Requirement tested by System qualities test 1.31

32 Make V&V activities part of regular flow 4System demos include validation status towards compliance 4Include compliance concerns in Definition of Done (DoD) 4Development continuously verifies Iteration verification Iteration validation PI validation Release validation System demo Solution demo Feature Story Story NFR Feature 1.32

33 Iteration verification DoD includes product and process concerns Feature Story design review Story DoD Peer review Delivered and baselined Update automated tests Actions recorded & resolved QA and Security review Solution Intent updated Agile Team Continuous V&V for Team s work 4 Peer review all engineering artifacts 4 Deliver all artifacts to the common repository 4 Reflect any discoveries or decisions in Solution Intent 4 Test Solution with story acceptance tests (ideally automated) 4 Ensure sufficient automation and/or compliance representatives to not bottleneck teams Primarily development team responsibilities 1.33

34 Iteration validation System Demo Evaluate full system increment 4 Regression test all functional stories, NFRs, and feature acceptance tests Feature Story Design review Feature DoD Pass all acceptance tests UAT/Customer signoff Quality audit Safety/Sec/etc. review Peer review Delivered and baselined Update automated tests All actions recorded All actions resolved Solution Intent updated 4 Tested on end-to-end test environment 4 User/Product Owner validation 4 Update V&V tests 4 Generate compliance docs and check progress towards acceptance Development teams, system team and program shared V&V responsibilities 1.34

35 PI validation Consider compliance in planning Innovation and Planning (IP) Solution Demo Program Increment Product Mgt Customer /user Evaluate full PI system increment 4 Regression test all functional stories and feature acceptance tests 4 User, PO, and PM validation 4 NFR tests 4 Assess progress of compliance information and documentation Program-based V&V responsibilities (May require handoff to independent V&V) 1.35

36 Release validation Fitness for purpose 4 Final validation 4 Qualification including installation and operation 4 User acceptance testing Supporting documentation 4 Installation, user, and other 4 Compliance evidence Releasable Solution Solution Increment System Increment Team Increment 1.36

37 Summary 1.37

38 Summary Build compliance into the system with SAFe 4Traditional QMS locks in waterfall development, and all its challenges large batches, late feedback, slows flow 4SAFe provides lean-agile mechanisms to achieve compliance 4Results in a leaner, more efficient development which produces better outcomes, higher quality 1.38

39 Here s how 4Build the solution and compliance incrementally 4Organize for value and compliance 4Build quality and compliance in 4Apply continuous verification and validation 1.39

40 High Assurance Training Opportunity! Post-Conference Training Friday, October 6

41 Questions? 1.41

42 Backup slides 1.42

43 Apply fast, cadence-based learning cycles Iterative development enabled continuous V&V and Compliance 4 Improve learning efficiency by decreasing the time between action and effect 4 Provide incremental feedback on our ability to meet both functional and compliance objectives The shorter the cycles, the faster the feedback P D A C P D A C P A P D A C P D A C D C P D A C P D A C Plan Act P D A C Do Check The Lean Machine: How Harley Davidson Drove Top-Line Growth and Profitability with Revolutionary Lean Product Development Dantar P. Oosterwal 1.43

44 Set-based design enhances quality Do we need this. It s a regulatory deck, not quality deck Too early design decisions assume a point solution exists, and can be built right the first time Point solution Requirements complete Design complete Planned deployment Insufficient time, $ to adjust Set-based Optimal solution Set-based design: Gain knowledge first, make decisions later 1.44

45 Adaptive high beam headlamp example Feature Detect oncoming traffic to dim headlamps Peer review Quality audit Customer sign-off Right process implements Requirements Model Test Model System 3.2 Headlamps Dimming distance Adaptive headlamps tests Feature acceptance test Right product Feature acceptance test Objective Evidence Regulatory Functional System Test Plan 1.45

46 SAFe s Lean-agile principles produce better compliance DELETE? outcomes #1 - Take an economic view #2 - Apply systems thinking #3 - Assume variability; preserve options #4 - Build incrementally with fast, integrated learning cycles #5 - Base milestones on objective evaluation of working systems #6 - Visualize and limit WIP, reduce batch sizes, and manage queue lengths #7 - Apply cadence, synchronize with cross-domain planning #8 - Unlock the intrinsic motivation of knowledge workers #9 - Decentralize decision-making 1.46

47 Synchronize development and compliance with PI Planning Future product development tasks can t be pre-determined. Distribute planning and control to those who can understand and react to the end results. Michael Kennedy, Product Development for the Lean Enterprise 4 All stakeholders face-to-face (safety, DELETE security, compliance, ME if we agree etc.) on 4 Balance investment in new functionality replacing with compliance with the and previous other concerns slidec 4 Requirements and design emerge through collaboration 4 Shared understanding of important stakeholder decisions 4 Teams create and take responsibility for plans PI 1.47

48 SAFe builds quality and other practices into the system Assure every increment of the solution reflects quality standards Software 4Continuous integration 4Test-First 4Refactoring 4Pair-work 4Collective ownership Hardware 4 Exploratory early iterations 4 Model Based Systems Engineering (MBSE) 4 Set-Based Design 4 Frequent, system-level integration Make statement that Agile builds this stuff in. We build quality in as part of flow. We want to do the same for compliance activities. 4 Design verification 1.48

49 Drives frequent solution integration with a solution demo 4Frequent solution integration and testing provides the best objective evidence 4Early end-to-end solution enables earlier V&V, customer signoff, feedback on regulatory approval 4View system demos as compliance/audit demos Reducing risk. Striving to make the last sign-off step boring. What was a huge, extended event to a nonevent. System Teams AGILE RELEASE TRAIN AGILE RELEASE TRAIN AGILE RELEASE TRAIN Program Increment Full or partial integration during the PI Full solution integration Program Increment Value Steam Program 1.49

50 Accelerate feedback continuous, automated testing Create tests that verify the solution s functionality as well as Need the a compliance better figure to standards and audit-ability Look at Wikispeed 4Automate tests in the same iteration as the functionality 4Include compliance tests safety, security, performance,quality, etc. 4Invest in automated testing infrastructure to improve flow 4Actively maintain test data under version control ü Func test Sec test ü Perf test QA test û Func test Progress Functional test automation ü ü ü ü ü Func test Sec test Perf test QA test Func test Done Building functionality Compliance test automation 1.50

51 Support continuous V&V DELETE THIS Automated and Manual SUPPORTING DEVELOPMENT Automated Functional tests Feature/Capability acceptance tests Enabler acceptance tests Story acceptance tests Unit tests Component tests BUSINESS-FACING Q2 Q1 Q3 Q4 TECHNOLOGY-FACING Acceptance tests Scenario tests Exploratory tests User acceptance tests Alpha and beta tests System qualities tests Performance and load Security Other NFRs Enabler tests Adapted from Brian Marick, Crispin and Gregory Manual and Automated CRITIQUING THE SOLUTION Tools 1.51