The Macro Process Is the Micro Process

The Macro Process Is the Micro Process Israel Gat, Director and Fellow (With many thanks to Murray Cantor, Tom Grant and Paul Ryan) IEEE Computer Society Symposium November 12, 2014

Bio Areas of research & consulting: Agile methods Devops Software governance Technical debt Technical Due Diligence Major products delivered: BMC Performance Manager/PATROL Microsoft Operations Manager Tivoli Smart Handheld Device Manager EMC Cellera Digital s NetView Nixdorf 8890 Books: The Concise Executive Guide to Agile Sample accolades: Winner, 2006 Innovator Award [Application Development Trends, May 2006] "When I deal with technical debt issues, I refer to Israel Gat regularly. His approach is the only one I've found that actually works [Director, Verisk Health] Nearly three times faster time to market than industry average one quarter the expected number of defects based on team sizes and schedules. [QSM Study, August 2007] The change you bought to BMC with Agile is the single largest change to the development model that I have ever witnessed in my almost 20 years at BMC. [Director, BMC Software] 2

Agenda Faking It Recipe for Software Development in our Era Single-Piece Batch The New Semantics 3

Part I: Faking It 4

Faking It Parnas and Clements in their 1986 paper A Rational Design Process: How and Why to Fake It we will never find a process that allows us to design software in a perfectly rational way the good news is that we can fake it if we agree on an ideal process, it becomes much easier to measure the progress that the process is making. 5

Why is the Software Development Process so Difficult that We Need to Fake It?! When an individual or a team solves a problem, they engage in four activities: Scoping. Ensuring they fully understand the problem. Designing. Developing an approach to solving the problem, usually using some sort of sketch or diagram. Implementing. Carrying out the design. Verifying. Confirming that the solution actually solves the original problem. It is important to understand that these are activities, not phases. An activity is something you do to reach an outcome. Phases are the steps in the lifecycle that mark the project s progress. Phases are not strictly tied to problem-solving activities since the activities often span the phases. [Murray Cantor] 6

Grady Booch s Two-Tier Process Structure Phases Process Workflows Inception Elaboration Construction Transition Business Modeling Requirements Analysis & Design Implementation Test Deployment Supporting Workflows Configuration Mgmt Management Environment Preliminary Iteration(s) #1 #2 #n #n+1 #n+2 #m #m+1 Iterations Source: Grady Booch, Rational Software

The Macro Process vis-à-vis the Micro Process The macro process represents the activities of the entire development team/value stream in terms of content and time Content: Requirements Analysis & design Implementation Test Deployment Time: Milestones to ensure quality or maturity, not dates per se The Micro Process represents the technical activities of the development team Traditionally analysis and design, e,g, in the Booch method 8

The Macro Process vis-à-vis the Micro Process Strong separation of concern between the two processes Macro Process: choice of lifecycle style: Waterfall, Iterative, Agile, etc. Micro Process: choice of technique, e.g. Object Oriented The Macro Process drives the Micro Process by prescribing products and activities that enable: Assessing risk Early corrections to the Micro Process 9

The Flow Problem: Handoff from One Phase to the Next One To represent reality, the macro process must capture wait periods between phases Like it or not, your artifacts are waitinggggggggggggggggggggggggggggggggg VSM has been used as a supplementary tool to that purpose, but in general it has not been explicitly incorporated in the macro process Only way to do so is by incorporating artifact wait states in the process definition See http://www.cutter.com/offers/devintell.html As various studies by Don Reinertsen have illustrated, the waiting periods between phases might be more important (in terms of flow than) the phases themselves 10

Part II: Single-Piece Batch 11

Single-Piece Batch Enables Optimizing Flow 12

The Process is Managed through Software Development Analytics Before After Source: Murray Cantor

By the Way, It All Applies to Devops The whole business of deployment as a separate activity/discipline is an extremely unfortunate consequence of archaic organizational structures All you need to do is handle deployment as ordinary steps/tasks in your single-piece batch system 14

Part III: The New Semantics 15

You Win on Flow, You Lose on Grouping Substance The very nature of the phases concept is transformed: Phase gates Flow optimization levers (WIP limits, etc.) Question: How do we to capture the user mental model? Think of the Table construct in a text editor Not understanding it in an early phase has a cost for flow, value delivered By analyzing the amount of investment in understanding the user mental model needed to increase the reliability of value delivered, analyzed at the micro level, we can reach macro level conclusions Think of what it means: we de-facto reintroduced the phase construct through new techniques 16

Question: When is Conception Completed in the Era of Continuous Deployment? Phases Process Workflows Inception Elaboration Construction Transition Business Modeling Requirements Analysis & Design Implementation Test Deployment Supporting Workflows Configuration Mgmt Management Environment Preliminary Iteration(s) #1 #2 #n #n+1 #n+2 #m #m+1 Iterations

Answer: Three Kinds of Development More Descriptive More Predictive Source: Murray Cantor

Part IV: Recipe for Software Development in our Era 19

First Step: What Kind of Project is on Your Hands? Are you doing: A new platform? New features on existing platform? Maintenance and small change requests on existing platform? Each kind requires different kind of analytics Moreover, for a new platform you will need to resurrect some form of the Elaboration phase to ensure just enough architectural runway to enable future small changes and new features will add up Appropriately enough, we call it Agile RUP 20

Second Step: Minimize Number of Roles; Maximize Flexibility End User The Business Customers Domain Experts Developers & Testers End User Feature priorities, scope Purchase convenience Product/ feature feasibility Quality and proper functionality The Business Feasibility Create standards Process requirements Feasibility Source of revenue Customers A market Products and services Create standards Compliance with standards Source of revenue Domain Experts Range of product variation Workplace well-being Need for standards Domain synergies and conflicts Constraints on technology Developers & Testers Requirement clarification Workplace well-being Advice on delivery process Guidance, APIs, poka-yoke Clarification of how existing code works Read down the columns to see what the roles contribute to the value stream; rows indicate the roles to whom value is provided. Source: Coplien & Bjornvig, Lean Architecture for Agile Software Development, Wiley, 2010. 21

Alignment Low Alignment High Third Step: Build on the First Two Steps to Attain Autonomy through Alignment High Autonomy w/high Alignment is Possible Alignment vs. Autonomy is a false dichotomy Alignment leads to autonomy My job is to provide the alignment: Overall team goals Overall business goals Top Level product strategy Your jobs are to creatively work to achieve these goals by innovative problem solving Silos Everyone gets their thing done, not necessarily overall success. Autonomy Low Salt Mines MIcromanagement. Innovati on Everone knows all of the goals and works to achieve them though problem solving. Autonomy High Anarchy This is not the meaning of agile (stolen from http://vimeo.com/85490944) Slide used with the kind permission of Mr. Paul Ryan 22

QUESTIONS? 23

Thank You! Israel Gat igat@cutter.com @agile_exec 24