system design & management

Transcription

1 system design & management Agile Project Dynamics for Aerospace and Defense Technologies Plus Lessons for Other Sectors Firas Glaiel Software Engineering Tech Staff Integrated Defense Systems Raytheon

2 Agenda Context Agile Genome Framework Agile Project Dynamics (APD) Model Experiments with the APD Model Insights and Next Steps 2

3 CONTEXT 3

4 Encouraging Reports About Agile The Problem Traditional large-scale software engineering: plan-oriented, process-heavy, gated, slow, costly, risk averse. Focus on risk management iron triangle of cost-scopeschedule rather than customer satisfaction. Requirements often overtaken by events (technology change, mission needs change, leadership change). Not new C2 Acquisition Study recommended buy a little, test a little to minimize overall exposure and risk. Agile as a Solution? Published studies of Agile show: Cost decreases of 5% to 61% Development time decreases of 24% to 58% Cumulative defects decreases of 11% to 83% BUT (Important Questions) What do people really mean by Agile? Why the variations in results? What are management acquisitions options and implications? 4

5 Generic DoD Acquisition Process Technology Change Technology Change Technology Change Technology Change Technology Change Technology Change User need identified Personnel rotation Personnel rotation Personnel rotation Product Delivered Personnel rotation *Reference: Mar 2009 DSB Report Personnel Rotation about every 3 years Technology Changes about every 2 years Not Efficient or Effective For DoD Acquisition Source: Acquisition of Information Technology briefing by Ronald Pontius, Director, Command and Control, OUSD (Acquisition, Technology & Logistics), March 20,

6 AGILE GENOME FRAMEWORK 6

7 Survey of Agile Methodologies Source: Forrester Report, July 2010: Water-Scrum-Fall Is The Reality Of Agile For Most Organizations Today 7

8 The Agile Genome Feature Driven Iterative & Incremental (Workload Management) Customer Involvement Micro-Optimization Refactoring Continuous Integration (Development Management) Team Dynamics (Soft Factors) The essence of Agile captured in these 7 Genes 8

9 Gene 1: Feature driven Break up of project into manageable pieces of functionality: sometimes called features, stories, use cases, or threads As opposed to functional decomposition The feature is a small, client-valued function. The final software product is the collection of all features. Working software over Comprehensive documentation Working software is the primary measure of progress

10 Gene 2: Iterative-Incremental Architecting system as a set of features is what allows for incremental and iterative development. Product Backlog Feature 4 Feature 5 Feature 6 Feature 7 Release Backlog Sprint Planning Sprint Backlog Feature 4 Feature 5 task task task task task task Product Feature 1 Feature 2 Feature 3 Feature 5 Feature 4 Sprint 1 Increment Sprint 2 Increment Release x.x Feature 8 Feature 9 Feature 10 Daily Scrum Working software over Comprehensive documentation Deliver working software frequently, from a couple of weeks to a couple of months.

11 Waterfall (BDUF) Walk-through Design Module A.1 Implementatio Module n A.2 Implementatio Module n A.n Implementation UI Subsystem Impl User Interface Requirements ATM Requirements ICD/IRS DB Requirements Design Module B.1 Implementatio Module n B.2 Implementation Module B.n Implementation DB Subsystem Impl Integrated ATM S/W ICD/IRS HW Controller Requirements Design Module C.1 Implementatio Module n C.2 Implementation Module C.n Implementation HW Controller Impl Analysis Requirements Specs Design Code U.T. SWIT SYS Int 1 1

12 Agile Walk-through ATM Requirements 24 hrs Feature 1 Check Balance Feature 2 Withdraw Cash Feature 3 Deposit Check Sprint Planning Feature Feature 11 Check Check Balance Feature Feature 22 Withdraw Cash Cash 3-4 weeks Sprint Shippable Product Increment Feature 4 Deposit Cash Feature 5 Transfer Sprint Backlog Product Backlog

13 Gene 3: Refactoring An incremental process can produce sub-optimal results compared to a waterfall model where complete up-front design is optimized for a full-featured release. Many agile methodologies (in particular XP) consider refactoring to be a primary development practice. Refactoring needed to pay off the technical/design debt Refactoring has the disadvantage that it takes extra effort and requires changing the software. Any change has the potential to reduce maturity and stability. That s why refactoring is usually paired up with Continuous Integration.

14 Gene 4: Micro-Optimizing Optimizing : teams are empowered to modify aspects of the process or dynamically adapt to changing circumstances. Micro : small improvements and variable changes are made frequently and as needed. E.g. team gradually figures out how many story points they can handle in a time-box. Double-loop learning: an individual, organization or entity is able, having attempted to achieve a goal on different occasions, to modify the goal in the light of experience or possibly even reject the goal.

15 Gene 5: Customer Involvement -Feature demos help identify rework in completed tasks: higher rework discovery rate. -Higher frequency of user feedback reduces requirements uncertainty Why Software Projects Fail

16 Gene 6: Team Dynamics Schedule pressure at the end of each iteration leads to higher work intensity. Short iterations means schedule pressure happens more frequently Higher frequency of communication increases the rate of gaining experience and reduces time to discover rework. Practices such as pair programming reduce defects (increased fraction correct and complete) Experience Gain / Learning : Project learning Process learning People learning

17 Gene 7: Continuous Integration Software Configuration Management practices E.g. sync-n-stabilize; merge early & often Build/delivery automation. Automated Testing - software does the testing that would otherwise be done manually. Once tests have been automated, they can be run quickly and repeatedly

18 AGILE PROJECT DYNAMICS (APD) MODEL 18

19 Basis for APD Model Agile Genome 19

20 Model of Getting Work Done Traditional View of Task Accomplishment PEOPLE PRODUCTIVITY DEFECT RATE Rework added to backlog WORK TO BE DONE REWORK TO BE DONE WORK BEING DONE REWORK CYCLE WORK DONE Some work done is incomplete or incorrect QA, testing, and user experience expose problems UNDISCOVERED REWORK Defects accumulate but are unseen. Errors-upon-errors second order effect of unseen defects 20

21 <Current Release ID> <TIME STEP> High Level SD Model View Developers needed to complete on schedule Schedule Pressure <Effective Sprint Duration> Staff Churn Rate <Switch for Waterfall> Percent of Team Change Per Sprint Effort Remaining in Sprint Time to Perceive Working Sched Pressure Harder at End of <Number of Team Sprint Meetings per week> <Switch for Waterfall> Max Work Intensity Work Intensity Normal Work Intensity <Staff> Nominal Time to Gain Experience In Agile Inexp Staff Churn Exp Staff Churn New Staff Experienced Initial Staff Experienced Nominal Time to Initial Staff Gaining Staff Gain Experience Inexperienced Experience Staff In Waterfall Time to Gain Experience Effective Staff Based on Experience Effect of <Switch for Experience on FCC Waterfall> Relative Experience <Effect of PP on Experience Gain> of New Staff Effect of Experience Staffing on Productivity Staff Sprint Backlog ME STEP> ss> <Productivity> Sprint Rework Discovery Burn Out Sprint Work Being Accomplished Sprint Rework Generation Work Performed in Project Team Dynamics Undiscovered Rework in Sprint Sprint Work Rate <Sprint In Progress> Haste Makes Watse Fraction Correct and Complete Effective Staff based on Work Intensity Time for Pressure to Effect FCC Sensitivity for Effect of Intensity on FCC Lagged Work Intensity for FCC Effect of Intensity on FCC Effect of MO on FCC Effect of MO on Pdy Effect of MO on Rework Discovery Allow Micro Optimization Sensitivity for Effects of MO Improvement in FCC based on Sprints Completed Improvement in Pdy based on Sprints Completed Improvement in Rework Discovery based on Sprints Completed <Sprint End Counter> one sprint <TIME STEP> Sprint Size Decrease <Current Sprint ID> <Switch for Waterfall> <TIME STEP> Sprint Size Increase Ideal Sprint Size <Sprint End Event> Initial Sprint Size Sprints Per Release <Release Size> <Current Sprint ID> Sprint Size Gap at end of Sprint <Release Cycle <Effective Sprint Duration> Duration> <Sprint Backlog> <Time Left In Sprint> <Staff> Extra <Productivity> Bandwidth sprint size increase rate <Sprint In Progress> Micro Optimization <Release Size> Fraction of Release Work Completed <Release Backlog> <InRelease> Requirements Churn Rate Based on Customer Involvement Sensitivity for the Effect of CustInv on Requirements Churn Allow Customer Involvement Effect of Customer Involvement on Requirements Churn <Release Work Done Correctly> Tech Debt Accrued per unit of Work Allow Continuous Integration <Current Sprint ID> CI Setup Work Generation Rate <InRelease> Effect of Uncertain Requirements on FCC Effect of Customer Involvement on Rework Discovery Time Effect of Tech Debt on FCC Technical Debt Accrual Rate Time to set up CI Staff to set up CI Continuous Integration Setup Work Number of Tasks to Setup CI Environment <Nominal Productivity> CI setup work rate CI Environment Available Elimination of Requirements Uncertainty Based on Sprint Progress Max Effect of Requirements Uncertainty Max Effect of Customer Involvement on Rework Discovery Time Improvement in Rework Discovery Due to Customer Involvement <Fraction Customer of Sprint Work <Number of Team Believed Done Correct and Meetings per week> Complete> Involvement Technical Debt <Sprint Work Being Allow Accrual Accomplished> Refactoring Technical Debt Technical Debt Pay Off Amount of Work <Start of Release Cycle Event> Planning to pay Refactoring Work Down Technical Debt <Sprint Duration> Generation Rate <TIME STEP> <Sprint Size> Refactoring Agressiveness Planned Refactoring Allocation of Refactoring to Release Refactoring Work Effect of CI on Effect of CI on Rework Productivity Discovery Rate Effect of Automated Testing on Rework Discovery Effect of CM Environment on Pdy Effect of CM Environment on Effect of Test Rework Discovery Automation on Pdy Continuous Integration Level of Automated Testing Used CM and Build Environement Automation Level 21

22 A Closer Look at the Rework Cycle Productivity Effective Staff Product Backlog Work Transfer Release Backlog Work Transfer Sprint Backlog Work Being Done Total Work Perceived Complete Delivered to Product Rework Discovery Work Transfer Rework Discovery Rate Product Release Rate Rework Discovery Fraction Correct and Complete Undiscovered Rework in Release Defects Escaping into Release Undiscovered Rework in Sprint Rework Generation Rate Agile Genes influence the Rework Cycle through key factors (Productivity, etc.) 22

23 Technical Debt In Our Industry 10/1 23

24 Sub-Model for Technical Debt and Refactoring <Release Work Done Correctly> Effect of Tech Debt on FCC Tech Debt Accrued per unit of Work Effect on Project Technical Debt Accrual Rate How much debt are we accumulating? Allow Refactoring Technical Debt Accrual <Sprint Work Being Accomplished> <Start of Release Cycle Event> Technical Debt Technical Debt Pay Off Amount of Work When do we decide to refactor? Refactoring Work Generation Rate Allocation of Refactoring to Release <TIME STEP> Planned Refactoring Work Planning to pay Down Technical Debt Refactoring Agressiveness <Sprint Size> <Sprint Duration> Planned refactoring goes into Release 24

25 The Greatest Challenge Debt: 10/17/

26 But first, what is Technical Debt? 10/17/

27 Documentation 10/1 27

28 Obsolescence, COTS/FOSS 28

29 Obsolescence - platforms s 2010s 29

30 Cybersecurity Copyright 2016 Raytheon Company 30

31 Architectural Debt Courtesy of: 31

32 The Real Challenge: Understanding the Impact Source: 32

33 The OODA Loop of Technical Debt Management Observe Orient Decide Act 10/1 33

34 Accumulation of Technical Debt and Pay Down via Refactoring ,000 Technical Debt ,000 task task ,000 5, Time (Week) Time (Week) Technical Debt : Case w/o TDM 2 Technical Debt : Case w/ TDM 3 Technical Debt : Current Technical Debt Pay Off Amount of Work : Current 34

35 Management Dashboard Schedule Policy Levers Quality Cost Project Parameters 35

36 EXPERIMENTS WITH THE APD MODEL 36

37 Experiments: Test Parameters and Cases Project Size = 200,000 tasks (e.g. SLOC or other measures) Total Staff on Project = 20 people Nominal Productivity = 200 tasks per-week, per-person Nominal Fraction Correct and Complete = 80% Relative Effectiveness of New Staff = 20% Number of Releases (if Agile) = 1 release Sprint duration (if Agile) = 4 weeks Expected Cost= 200,000 tasks Expected Duration = 50 Weeks Case Name Feature Driven Continuous Integration Iterative/ Incremental Refactor Customer Involvement Micro- Optimize Pair Programming Case 1 Tradition al OFF OFF OFF OFF OFF OFF OFF Case 2 Scrum ON OFF OFF ON OFF ON OFF 37

38 Case 1 - Traditional 200,000 task 1 Dmnl 100,000 task 0.5 Dmnl 0 task 0 Dmnl Product Backlog Release Backlog Sprint Backlog Dev End Schedule (weeks) Schedule (Backlogs) Time (Week) Project Performance Development Effort (tasks) Time to Complete 65 weeks task task task Dmnl Defect Discovery (weeks) CPI SPI Quality (Defects in Product) 2,000 task 1 Dmnl 1,000 task 0.5 Dmnl 0 task 0 Dmnl Defects Dev Finished Time (Week) task Dmnl Cost (Development Effort Spent) task 400, ,000 0 Development Effort Time (Week) 38

39 Case 2 - Scrum 200,000 task 1 Dmnl 99,800 task 0.5 Dmnl -400 task 0 Dmnl Product Backlog Release Backlog Sprint Backlog Dev End Schedule (weeks) Schedule (Backlogs) Project Performance Development Effort (tasks) Defect Discovery (weeks) CPI SPI Time to complete 50 weeks Time (Week) task task task Dmnl Quality (Defects in Product) 4,000 task 1 Dmnl 2,000 task 0.5 Dmnl 0 task 0 Dmnl Defects Dev Finished Time (Week) task Dmnl Cost (Development Effort Spent) task 400, ,000 0 Development Effort Time (Week) 39

40 Other Variations 200,000 task 1 Dmnl Schedule (Backlogs) Multiple Releases 99,900 task 0.5 Dmnl -200 task 0 Dmnl Product Backlog Release Backlog Sprint Backlog Dev End Time (Week) task task task Dmnl There are many other variations of genes and policy variable combinations possible. 40

41 Impacts of Funding Fluctuations Staffing Data Rate Of Development 40 6, ,500 person task/week 3,000 1, Time (Week) New Staff : current_var Experienced Staff : current_var Total Staff : current_var Effective Staff Based on Experience : current_var Time (Week) Rate Of Development Funding Gap Ramp-up Tax Gap Tax Model also used to understand the impact of budget cuts and fluctuations 41

42 CONCLUSION 42

43 How the Agile Dynamics Model Helps What do people really mean by Agile? Agile Genome captures the essence of Agile 7 genes. APD models how the genes work individually and in combination to produce emergent system behaviors -- the enterprise/system properties are not a sum of the parts, but a product of their interactions. Why the variations in results? Model shows variations from different combinations of genes and exogenous factors. Some results are worse than traditional waterfall. Many are better. What are management options and implications? Even within a tightly constraining acquisition process, strategic use of Agile genes can be beneficial. 43

44 Insights Gained The developed product itself is part of the system under study, in addition to the people, processes, and tools and the interactions thereof. True Agility is not only reflective of the agility of the team or Leanness of the process, but also of the flexibility of system being developed. APD suggest that, to gain the full benefits of Agile, one cannot simply cherry-pick easy-to-adopt development techniques and hope to achieve improved project performance. 10/17/

45 Insights Gained (cont) Each practice/policy can have positive and negative effects. The trick: configure the system (program) to maximizes upside / minimize downside. ( Genetic Program Engineering ) Examples: Incremental development Technical Debt Refactoring Refactoring Productivity Loss Continuous Integration 10/17/

46 Next Steps Gather feedback and revise model elements. (The model evolves iteratively/incrementally). Operationalize the model by mapping model variables to available metrics of parameters, policy choices, and results of past projects. Validation: Test, calibrate, and update the model based on comparison of actual project data to model predictions. Transition validated and calibrated model for use in acquisition/development for strategic project management MITRE and MIT are working on validating the model against real cases and potentially extending the model. 46

47 THANK YOU For more info, see Agile Project Dynamics Whitepaper: Firas Glaiel Special thanks to contributors: Allen Moulton Prof Stuart Madnick and the MIT Interdisciplinary Consortium for Improving Critical Infrastructure Cybersecurity (IC) 3 47