Dr. Satish Thatte CEO, New Synergy Group. Kendall Park, NJ

Transcription

1 Turbo-Charging Agile Software Development with Lean Methods, Systems Thinking and High-Performance Teams Technical Colloquium Villanova University 1 November :30 PM 5:45 PM Dr. Satish Thatte CEO, New Synergy Group Kendall Park, NJ

2 Agenda Agile methods (such as Scrum) are enjoying growing momentum in software industry For customers: high value software meeting their needs For suppliers: Increased productivity of software producers without sacrificing quality; in fact, quality also is often improved Ability to deliver on tight schedule in a rapidly changing and uncertain business environment Agile methods can be turbo-charged with: Lean methods Systems thinking high-performance teams 11/2/ New Synergy Group 2

3 Challenges Facing Software-Intensive Industry Failure to meet the real requirements of users and customers Schedule overruns; unpredictable schedules Cost overruns Poor quality: too many defects, poor reliability, poor availability Low productivity, and Low Productivity x Quality measure These challenges have been with us for the last few decades. And they will continue to be with many companies: Enterprises continue to develop ever more ambitious and complex software systems User expectations continue to increase Fierce global competition increases pressure to develop and deliver new innovative software products, solutions and services in ever shorter cycle times in an increasingly volatile and unpredictable global business environment. 11/2/ New Synergy Group 3

4 What We Know about Software Development Processes What we knew in the 20 th century Right people and high-performance teams always trump processes and tools. But without processes and tools, their productivity is reduced. What we learned in the 2000 to 2009 decade Improved software processes and project management (Agile/Scrum, Lean methods) are important, but may not be enough. Where we are heading in the 2010 decade Solutions addressing systemic problems are often more enduring than those accomplished by improving software project management, or by changing the software development process alone. Create and sustain highly motivated, peak performance teams Huge untapped potential 11/2/ New Synergy Group 4

5 Project Three Engines for the 2010 Decade: Processes, Organizational Systems, High-Performance Teams. High Perf Teams Agile and Lean processes Organizational Systems Thinking Effect of booster rockets is multiplicative if synchronized and managed properly 11/2/ New Synergy Group 5

6 Journey to Agility: Scrum is the Leader Ad hoc Scrum is the market and mindshare leader Simple framework, which is extensible and customizable. 10% projects following Scrum, and 30% following one or more agile methods Forrester Survey, Jan Water Fall Iterative Agile Google, Yahoo!, SalesForce.com, Microsoft, Cisco, Intel, GE, Game Developers, even Church Management. 11/2/ New Synergy Group 6

7 Employ Iterative, Feature-Based Delivery Feature-based: Build features of the final product in each iteration. Customers neither understand nor care about engineering tasks; they understand and care about features Features of a smart phone app, navigation system of a car Iterative: Build a partial version of a product and then expand that version through successive short time periods of development followed by reviews, feedback and adaptations. Incremental: Build upon these partial products that will build functionality incrementally in successive iterations. Time-boxed: Produce a result meaningful to the customer within a certain period of time (time-box) in each iteration. 11/2/ New Synergy Group 7

8 Example: Smart Phone Product Basic phone operations: Make call, Receive call Set-Up: Sound, Display, Phone, Call, Blue Tooth Contact Management: New, List, Group, Speed Dial, Search Call Log Management: Dialed, Received, Missed, All Messaging: NewText, InBox, Voice Mail, MobileIM, Chat Tools: Volume, Calendar, WorldClock, AlarmClock, Notes Music: AllSongs, PlayList, Artists, Genre, Album, Settings, Play Camera: Picture, Album, Settings 11/2/ New Synergy Group 8

9 Concurrent Engineering Sequential Engineering Realistic Waterfall vs. Realistic Scrum Cadence Waterfall (Release 1.0): 6 months.. delay.. 9 months Analysis Design Code Test Basic phone Set-Up Tools Basic phone Set-Up Tools Basic phone Set-Up Tools Agile / Scrum (Release 1.0) Basic phone Set-Up Tools Sprint 0 Architecture Framework Sprint 1 Basic phone Sprint 2 Set-up Sprint 3 Tool Set Hardening Sprint 4 (3 weeks) Analysis Analysis Analysis Analysis Analysis Design Design Design Design Design Code Code Code Code Code Test Test Test Test Test 11/2/ New Synergy Group 9

10 Traditional vs. Agile Projects Waterfall Agile The Plan creates Cost and Schedule estimates The vision and values create feature estimates Constraints Features Cost Schedule Plan- Driven Vision and Value Driven Estimates Cost Schedule Features 11/2/ New Synergy Group 10

11 Scrum is a Minimalist 3x3 Framework Three Roles Three Artifacts Three Ceremonies Product Owner Product Backlog Sprint Planning ScrumMaster Sprint Backlog Daily Scrum Scrum Team Burn-Down Charts Sprint Review This minimalist framework is great to get started. It works for pilots and single-team projects with all members collocated. Scalability: It is essential to extend this framework for larger projects 11/2/ New Synergy Group 11

12 . Scrum Onion : Looking into Future Vision Roadmap Release Sprint Daily Scrum Frequent Primary Responsibilities of Product Management & Owner: Product Vision and Roadmap Release Plan Prioritized Sprint backlog Primary Responsibilities of ScrumMaster Manage the Agile Scrum process Daily Scrums Impediment removals Primary Responsibilities of Scrum Team: Frequent Tasks: Design, Development, Testing, Daily Builds, Continuous Integration 11/2/ New Synergy Group 12

13 Scrum Method. Daily Scrums Release Backlog Sprint Backlog Time-boxed Sprint Velocity Potentially Shippable Product, Sprint Review and Retrospective Product Backlog 11/2/ New Synergy Group 13

14 Scrum Method. Daily Scrums Time-boxed Sprint Release Backlog Sprint Backlog Velocity Sprint 0 Rel. Sprint 1 Rel. Sprint N+1 Rel. = GA Rel. Product Backlog 11/2/ New Synergy Group 14

15 History and Key Principles of Lean Thinking and Methods Lean or Lean Thinking is the English term given by MIT researchers to describe the system now known as the Toyota Way Lean refers to Lean production, as opposed to Mass production Lean does not mean lean and mean or fire and downsize Competing not on economies of scale, but on the ability to adapt, minimize inventory and work in small cross-functional teams Measure of success must be related to the cycle time of Concept to Consumption to Value Creation Remove bottlenecks to faster throughput of value to customers rather than local optimizations (which try to maximize worker or machine utilization) Analogy: Focus on the baton instead of utilization of runners in a relay race Outperform the competition by a focus on short cycles, small batches and queues Most errors and wastes are due to the organizational systems within which people work rather that with individuals. 11/2/ New Synergy Group 15

16 Value and Waste in Processes Value: The time periods of actions while developing the product that the customer is willing to pay for. Value is in the eyes of (external) customer. Put yourself in customer shoes to get better appreciation Waste: All other time periods that do not add value, but consume resources, is waste. Waste comes from: Waiting in visible and invisible queues (Requirements, Design docs, Defects) Bottlenecks (Single experts who get pulled out for every crisis) Hand-offs (in sequential waterfall process) Wishful thinking (we have plenty of time left to make up) Undo-Redo cycles (due to poor communication, poor understanding, etc.) Information fragmentation (due to poor information or tool integration) Multi-tasking, context-switching (thrashing) workers Value Ratio = Total value time / Total lead time (concept to consumption time) Value ratio is shockingly low (less than 7%) in most product development organizations! [Source: Larman & Vodde, 2008; Poppendiek, 2006] There is nothing so useless as doing efficiently that which should not be done at all. -- Peter Drucker 11/2/ New Synergy Group 16

17 No-Value Actions in Software Development: I No-Value Actions Examples and Notes Overproduction of features Waiting in queues or Work-in-Progress queues Hand-offs Impedance mismatch Reinvention Features the customer does not want or rarely uses (45% features never get used; and 20% features get used 80% of the time Standish Group reports); Detailed requirements that are not going to be implemented in the next Sprint; Duplication of code or data Waiting for clarification, reviews, approvals: Requirements, design, test cases One team depending on another team for a component QA team waiting for defect fixing; Development team waiting for regression testing Requirements written, but not coded; Software coded but not tested; Defects logged but not fixed Requirements hand-off from an analyst to a developer; Code hand-off from a developer to a tester Interface between Agile and non-agile projects is awkward and imposes cost and overhead; needs to be removed quickly by making most/all projects go agile Redeveloping a component another developers has already developed 11/2/ New Synergy Group 17

18 No-Value Actions in Software Development: II Non-Value Adding Actions Context switching and interrupts Under-utilized talents of people (skills, insights, ideas, suggestions) Information fragmentation Wishful thinking (ex. the correctness of plans, estimates, specifications) Temporary waste given the current limitations Examples and Notes Interrupts to handle hot defects, hot demos; Multi-tasking and context switching across 3 or 4 projects; Partially allocated developers/testers across many projects People pigeon-holed or narrowly specialized in single-function skills Ex. UML Modelers, Performance/Load Testers, Flash/Ajax programmers, etc.) Information in many separate documents, instead of a central wiki Same information in multiple tools (Defects in Agile tool and a specialized defect tracking tool); Lack of tool integration The estimate cannot increase; we want a certain estimated effort, and not the one you (workers doing the work) have estimated! We are behind schedule, but we have plenty of time (9 out of 12 months), so we will somehow make it up later! Developers cannot avoid creating defects (which is a waste); reduce defects through test-driven development, and test and fix defects in small sprints 11/2/ New Synergy Group 18

19 Flow Management Make value flow without delay to the customer Value in products, services, software, information, decisions Reliable cadence of delivering sprint releases every n weeks No or minimal Queues of Work in Progress (WIPs) Minimize backflows (re-work trying to clean up the mess) Flow is perfection challenge (probably never achieved 100%; but it is a very important goal to strive) Zero waste by reducing batch size, variability, WIP, queues and delays, and other waste Continuous flowing delivery of value These attempts also help the goal of continuous improvement (Kaizen) 11/2/ New Synergy Group 19

20 Cycle Time vs. Capacity Utilization and Batch Sizes Cycle Time L: Large, M: Medium, S: Small Work Batch Sizes L M S Capacity Utilization Cycle Time = Queue time + Service time As capacity utilization increases, queue time increases non-linearly Large batches of work (ex. features) have a nonlinear impact on queue and cycle times Only batch size is under our control, as people are almost 100%+ loaded Decompose large features into smaller sub-features no feature taking more than ¼ N staffweeks of effort for N-week sprint 11/2/ New Synergy Group 20

21 Queue Management Techniques Eradicate or reduce queues wherever possible by changing or redesigning the organizational systems. Vanquish WIP queues by moving from serial to concurrent development. Avoid attempts to reduce queues by increasing multi-tasking or resource utilization rates. This will increase average cycle time, not reduce it! Develop eyes for seeing hidden queues invisible queues (soft documents, files, code in database) and eliminate them Software equivalent for Kanban boards for distributed teams 11/2/ New Synergy Group 21

22 Kanban Agile Development Flow: Set WIP Queues Limits Features with Design in Progress Features with Design Done Features with Development in Progress Features with Development Done Features with Test in Progress Features with Test Done Features with Defects fixing in Progress Flow Cross-functional, self-organizing team smoothens the flow by rebalancing WIP queues based on WIP Queue Limit = 3 (in this example) Test team needs more resources for testing, and Developers need for fixing more defects Features with Defects Verification Progress 11/2/ New Synergy Group 22

23 SPRINT BACKLOG Kanban Agile Development Flow: Set WIP Queues Limits Minimize end-to-end development cycle time by minimizing waste, delays and waits. Stop Starting new features, Start Finishing features in progress Pull Features with Design in Progress Features with Design Done Features with Development in Progress Features with Development Done Features with Test in Progress Features with Test Done Flow Do work only in response to pull, based on priorities in the Sprint Backlog Combine prioritized sprint backlog with lean flow of value Features with Defects fixing in Progress Features with Defects Verification in Progress 11/2/ New Synergy Group 23

24 Comparison between Agile/Scrum vs. Lean Agile/Scrum Lean Time-boxing Yes No Reduce end-toend feature cycle time No; a feature may consume all weeks of a spint (in pure Agile process) Flow No Yes Visibility Validation and Verification Optimization and scalability Control mechanism Information Radiators: Burn-up and Burndown charts, and many other types Test-driven development, Acceptance tests as part of requirements Refactoring, and emergent behavior, Scrum of Scrum and Meta Scrum Daily Scrum commitment by individuals, and Sprint commitment by Scrum team Yes; features are leveled to reduce average cycle time; focus on finishing features in progress before opening new features Visible WIP queues, Kanban boards Do it right the first time, Find the root cause and fix it fast System of systems (systems thinking); Optimize the Whole (not parts) WIP limits 11/2/ New Synergy Group 24

25 Introduction to Systems Thinking A system is a set of interacting or interdependent components or subsystems forming an integrated whole. A system is a subset of reality that is the focus of analysis. The key characteristics of a system are: Purpose or function: Usually deduced by observing system behavior, which involves inputs (information or material or energy), processing, and outputs (information or material or energy). Structure: Defined by components and their composition. A system involving people has values, principles and people as its components, as well as processes and business workflows. Interactions: Various components of a system have functional as well as structural relationships among each other allowing them to interact. Note: We are not talking about software system to be developed, but systems in general specifically organizational systems. 11/2/ New Synergy Group 25

26 Enterprise as a System of Six Sub-Systems 2-Nov New Synergy Group 26

27 System Models A system must be represented in some form in order to communicate about it and also to analyze it. A model is used to represent a system. A model captures only the key aspects of the system A map is a model of the geographical territory it represents. Causal Loop Diagrams and Stock-and-Flow Networks represent models or abstractions of real systems Causal Loop Diagrams (CLDs) Great for understanding and reaching consensus, but are qualitative. They cannot be simulated Stock-and-Flow Networks (SFNs) More complex and more effort They can be simulated for quantitative analysis and what-if experiments 11/2/ New Synergy Group 27

28 CLD for Relationship between Release Cycle Time and Number of High Priority Features Notation Cause and Effect variables are shown in rectangular boxes, with links showing causal relationships S (Supports): If the cause increases, the effect increases above what it would otherwise have been, and if the cause decreases, the effect decreases below what it would otherwise have been. R: Reinforcing (positive) feedback loop 11/2/ New Synergy Group 28

29 CLD for Relationship between Sprint Duration and Test Automation R: Reinforcing (positive) feedback loop B: Balancing (negative) feedback loop Notation Cause and Effect variables are shown in rectangular boxes, with links showing causal relationships O (Opposes): If the cause increases, the effect decreases below what it would otherwise have been, and if the cause decreases, the effect increases above what it would otherwise have been. S (Supports): If the cause increases, the effect increases above what it would otherwise have been, and if the cause decreases, the effect decreases below what it would otherwise have been. 11/2/ New Synergy Group 29

30 CLD for Relationship among Sprint Duration, Test Automation, Number of High Priority Features 11/2/ New Synergy Group 30

31 Synergies among Scrum, Lean, Systems Thinking Scrum advocates the Art of Possible Lean advocates Stop Starting, Start Finishing System thinking teaches us how to think holistically and optimize the whole, not sub-optimize the parts Systems thinking enables organizational systems that allow competent people do good work and produce outstanding results Scrum, Lean Methods and Systems Thinking are highly complementary and synergistic, and turbo charge agile development projects 11/2/ New Synergy Group 31

32 Transition From Agile-Lean-Systems Thinking to Your Work as Your Calling 7 +/ 2 people Scrum team Cross-functional, Self-organizing teams of selfdisciplined members Agile, lean, systems thinking methods and tools are by themselves not enough to develop motivated employees and high-performance teams Only high-performance teams of highly motivated people can produce great products and services, and build and grow great companies 11/2/ New Synergy Group 32

33 The Surprising Truth about What Motivates Us Let us watch this video 11/2/ New Synergy Group 33

34 Motivated Workers to High-Performance Teams Three essential elements for highly-motivated individuals Autonomy: The desire to direct our own lives Mastery: The urge to get better and better at things that matters Purpose: The yearning to do what we do in the service of something larger than ourselves. Five essential disciplines for high-performance teams Meaningful common purpose Inspiring performance goals that flow from the common purpose Mix of complementary skills Strong commitment to a common process for getting work done Mutual accountability From high-performance employee teams to stakeholder teams Software engineers, QA engineers, Product Owners, ScrumMasters, Customer representatives, Customer support, Marketing, Legal, HR 11/2/ New Synergy Group 34

35 The Platform: Agile and Lean Methods, Systems Thinking, and High-Performance Teams 11/2/ New Synergy Group 35

36 Software Dev. Processes Turbo-Charged with the Platform: Agile and Lean Methods, Systems Thinking, and High-Performance Teams 11/2/ New Synergy Group 36

37 Agile, Lean and Systems Thinking Practices Product Management Product vision, Elevator stmt, etc. Project Management Time-boxing Domain & Feature Analysis Problem domain analysis Design and Development Architecture sprints Product roadmap Sprint planning Features cards Simple design, Refactoring Product backlog Release planning Release data sheet Release backlog, Value and Biz risk estimation Info radiators: Burn charts, Kanban charts,.. Scrum of Scrums, Meta Scrums Sprint backlog, Effort and Tech risk estimation Use cases, User stories, Level the work Non-functional requirements Prototypes, Mock-ups Continuous integration Feature-driven design & development Test-driven development QA Early test development Automated testing Automated builds and deployment (Automated) Acceptance Tests Pull, Kaizen Reduce WIPs, 5-Why s, CLDs Retrospective feedbacks, Flow Personas and Scenarios Agile data and documentation 37

38 Agile Scrum Templates Product vision Product roadmap Product elevator statement Product Release Data Sheet Features Design review Code review Test cases Capacity calculations for each member of Scrum team Availability and capacity for Scrum team Prioritization of sprint backlog Steady workflow and load balancing for all sprint weeks Scrum team weekly planning Daily Scrum Ready-Ready checklist Done-Done checklist Sprint reviews Sprint retrospectives 11/2/ New Synergy Group 38

39 Turbo-Charging Agile Software Development with Lean Methods, Systems Thinking and High-Performance Teams Dr. Satish Thatte CEO, New Synergy Group Kendall Park, NJ