IN5140 Smart processes and agile methods in software engineering

Transcription

1 IN5140 Smart processes and agile methods in software engineering Lecture 2 October 2018: Lean and Agile Software Engineering Dag Sjøberg dagsj@ifi.uio.no IN5140/Lecture /Slide 1 Dag Sjøberg Clear relationship between process improvement measurements empirically/evidence-based decision making Lean agile -> It s about working smarter IN5140/Lecture /Slide 2 Dag Sjøberg 1

2 Relationship between lecture and book This lecture focuses on stuff in Chapters 2, 3, 4 (indirectly reduce cycle time) and 5 (indirectly) and 6 of the Larman & Vodde book. Emphasizes the relationship between Lean and Agile (more than the book does) IN5140/Lecture /Slide 3 Dag Sjøberg Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Agile principle 1: Satisfying the customer Flow Visualization Waste Causes of waste Agile principles 2 12 IN5140/Lecture /Slide 4 Dag Sjøberg 2

3 Systems thinking A system is an entity with interrelated and interdependent parts; it is defined by its boundaries and it is more than the sum of its parts Changing one part affects other parts and the whole system, with predictable patterns of behavior Positive growth and adaptation of a system depend upon how well the system is adjusted with its environment, and systems often exist to accomplish a common purpose (a work function) that also aids in the maintenance of the system or the operations may result in system failure [Wikipedia 2018] IN5140/Lecture /Slide 5 Dag Sjøberg Systems thinking is about understanding the effect of actions on the total system Global optimization As opposed to sub-optimization, which is local optimization System dynamics Consequences/Feedback loops Root cause analysis Usually several direct and indirect causes IN5140/Lecture /Slide 6 Dag Sjøberg 3

4 Technique for root cause analysis Ishikawa diagram* indicates relationship between (un)wanted effects and their possible causes Environment Humans Software Problem/Solution Methods Procedures Hardware *After the Japanese professor Kaoru Ishikawa also called fish bone diagram IN5140/Lecture /Slide 7 Dag Sjøberg Class exercise 3 minutes, groups of two: How can your studies at the University of Oslo become (even) better? Use Systems thinking: think large systems, feedback loops and root causes IN5140/Lecture /Slide 8 Dag Sjøberg 4

5 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Agile principle 1: Satisfying the customer Flow Visualization Waste Causes of waste Agile principles 2 12 IN5140/Lecture /Slide 9 Dag Sjøberg Purpose of Lean All parts of the organization should optimize their processes to achieve a common goal: To create value for the customer Value = what the customer wants and is willing to pay for IN5140/Lecture /Slide 10 Dag Sjøberg 5

6 Overall Lean principles Customer value Satisfying the customer Flow Visualization Avoiding waste Supporting change Project/Magic/Iron triangle IN5140/Lecture /Slide 11 Dag Sjøberg Lean production The Japanese school, primarily Toyota If failure discovered, stop the assembly line and find the reason for the failure instead of only collect and fix the failures in bulks (root cause analysis and root cause fix) Just-in-time (JIT) principle: don t produce before somebody demands it Tempo in production determined by pull from customer or next element in the production chain rather than push internally to produce as much as possible Removal of temporary storage Component-based production (same chassis, bumper, etc. on different car models). Production geared towards the customer, components can be outsourced to third-party vendors Continuous learning and improvement (Kaizen) Result: Toyota fewest failures and fastest production The most productive factories spent least resources on management and administration ( lean management ) IN5140/Lecture /Slide 12 Dag Sjøberg 6

7 Lean stems from Japanese car industry, but Now widespread in all kinds of knowledge-intensive industry and administration Large companies Hospitals Governments Universities Etc. IN5140/Lecture /Slide 13 Dag Sjøberg Much of today s process improvement happens under the Lean umbrella IN5140/Lecture /Slide 14 Dag Sjøberg 7

8 Similarities with the Norwegian/Nordic model Self governed (autonomous) team Learning, redundancy/job-rotation Work environment Quality of life Co-operation among management, trade unions and government Norwegian Hydro, Volvo and many more B. Gustavsen, T.U. Quale, B.A. Sørensen, M. Midtbø og P.H. Engelstad. Innovasjonssamarbeid mellom bedrifter og forskning den norske modellen. Gyldendal 2010 IN5140/Lecture /Slide 15 Dag Sjøberg Concepts & principles Lean versus Agile System development organization or department Other organization or departments Lean software engineering or agile development (agile methods) Lean (not agile ) Lean software engineering: focuses more on the context of the whole organization than agile does Some agile principles come from Lean Others are developed independently IN5140/Lecture /Slide 16 Dag Sjøberg 8

9 IN5140/Lecture /Slide 17 Dag Sjøberg How the manifesto has been interpreted and often practised [Janes & Succi 2012] 9

10 Agile Manifesto 12 principles* 1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. 2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. 3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. 4. Business people and developers must work together daily throughout the project. 5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. 6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. 7. Working software is the primary measure of progress. 8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. 9. Continuous attention to technical excellence and good design enhances agility. 10. Simplicity the art of maximizing the amount of work not done is essential. 11. The best architectures, requirements, and designs emerge from self-organizing teams. 12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. * IN5140/Lecture /Slide 19 Dag Sjøberg What are the relationships between the 12 agile principles and Lean? IN5140/Lecture /Slide 20 Dag Sjøberg 10

11 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Agile principle 1: Satisfying the customer Flow Visualization Waste Causes of waste Agile principles 2 12 IN5140/Lecture /Slide 21 Dag Sjøberg Agile principle 1: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. IN5140/Lecture /Slide 22 Dag Sjøberg 11

12 Who is the customer? One who pays for the system (sponsor), uses the system and/or gets value from the system Consider the whole value chain, including customers of the customer Customer may be internal or external (think total system) IN5140/Lecture /Slide 23 Dag Sjøberg Expectations of various stakeholders Customers: Do they prioritize everything, perfect and immediate delivery, or are they more realistic? Management: Consider overall goals, expect no surprises? Developers: Expect to build with top technical quality of technical, fancy system? Maintainers: Expect few bugs, understandable and well documented code? Other stakeholders (law makers, other systems) IN5140/Lecture /Slide 24 Dag Sjøberg 12

13 How to prioritize among stakeholders? Money rules Difficult to ignore the management IN5140/Lecture /Slide 25 Dag Sjøberg Flow Fundamental in Lean, less focus in Agile IN5140/Lecture /Slide 26 Dag Sjøberg 13

14 Total value stream systems thinking What is the customer s purpose with the system? How to ensure flow of a customer s request or product through the whole value stream? Value stream map of the organization to identify waste possibilities for optimizing the processes IN5140/Lecture /Slide 27 Dag Sjøberg Value stream mapping Example IN5140/Lecture /Slide 28 Dag Sjøberg 14

15 Product champion A person who has the business responsibility for the system (product) Ensures flow of the whole end-to-end process Must understand both business and technology processes IN5140/Lecture /Slide 29 Dag Sjøberg Visualization IN5140/Lecture /Slide 30 Dag Sjøberg 15

16 War room (obeya) A form of project management, where all those involved in planning (of a project) gather in one room Visual charts and graphs depicting: Vision/objectives Plan with milestones and progress Task board Metrics Timing or technical problems with countermeasures Reduces need for communication IN5140/Lecture /Slide 31 Dag Sjøberg Obeya war room Red: problems (issues) IN5140/Lecture /Slide 32 Dag Sjøberg 16

17 01/10/18 To help ensure flow, reduce waste IN5140/Lecture /Slide 33 Dag Sjøberg Waste To define waste, you must define what the customer is willing to pay for = Value Added Work What the customer is NOT willing to pay for Ø Increases the costs but not value = Non-value Added Work No value but is required by existing processes = Necessary but Non-value Added Work (for example, infrastructure) IN5140/Lecture /Slide 34 Dag Sjøberg 17

18 Waste definition Everything that requires resources and does not give value to the customer time work effort space equipment money IN5140/Lecture /Slide 35 Dag Sjøberg Waste examples 1. Overproduction 2. Wait 3. Transport. 4. Overprocessing 5. Queue/ inventory 6. Rework 7. Motion/ Task switching 8. Skills 9. Information scatter/loss 10. Wishful thinking 18

19 [Larman & Vodde, p.60-61] Also IT people should assess waste in business processes [Bell & Orzen 2011] Even IT people without domain knowledge may ask sensible questions IN5140/Lecture /Slide 38 Dag Sjøberg 19

20 Five important causes of waste in software development Complexity Scale instead of flow Separating decision making from work Wishful thinking Technical debt [Poppendieck & Poppendieck 2010] IN5140/Lecture /Slide 39 Dag Sjøberg Cause of waste 1: Complexity Software entities are more complex for their size than perhaps any other human construct because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into a subroutine open or closed. In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound. Likewise, a scaling-up of a software entity is not merely a repetition of the same elements in larger sizes, it is necessarily an increase in the number of different elements. In most cases, the elements interact with each other in some nonlinear fashion, and the complexity of the whole increases much more than linearly. [F. Brooks, No silver bullet: Essence and accidents of software engineering, IEEE computer vol. 20, ed, 1987, pp ] IN5140/Lecture /Slide 40 Dag Sjøberg 20

21 Be a bit sceptical to data from Standish Don t exclude features of great value even though they are rarely used Features should be considered from the combination of value and frequency of use IN5140/Lecture /Slide 42 Dag Sjøberg 21

22 Unnecessary complexity Waste category Examples Over processing Too complex business processes are automated Software development processes and best practices that give overhead Too much design too early Extra features Many features of the systems are not used Gold plating system design and performance Documentation is often not used Identify functionality that is not strongly needed Don t implement at all or defer to JIT (Just-in-time) Most code is rarely or never executed Don t measure individuals performance in LOC or Function Points (the code should be minimal) Improve work and business processes before they are automated IN5140/Lecture /Slide 44 Dag Sjøberg 22

23 Code size versus maintainability System A System B System C System D Average effort (hours) Java lines of code (KLOC) IN5140/Lecture /Slide 45 Dag Sjøberg Cause of waste 2: Scale instead of flow Traditional thinking that one needs to scale up to increase profit Efficient flow may be more important? Challenges: At present, there may be more focus on individual tasks than the whole value stream Always full utilization is considered optimal use of resources Too many tasks in parallel (WIP) IN5140/Lecture /Slide 46 Dag Sjøberg 23

24 Are you most efficient when you do many or few things in parallel? IN5140/Lecture /Slide 47 Dag Sjøberg Cost of context switching / multitasking [Clark and Wheelwright 1993] IN5140/Lecture /Slide 48 Dag Sjøberg 24

25 01/10/18 WIP (Work in Progress/Process) limit Kanban limits the number of work items in a given state Scrum limits the number of work items in a sprint, but not per state IN5140/Lecture /Slide 49 Dag Sjøberg Scrum board versus Kanban board Max WIP From: Kanban and Scrum - making the most of both by Henrik Kniberg and Mattias Skarin on Dec 21, 2009 IN5140/Lecture /Slide 50 Dag Sjøberg 25

26 Cause of waste 3: Separating decision making from work Managers must understand the work, that is, be able to ask relevant questions Should not define measurable goals without technical understanding, cf. Deming Are business processes and IT considered as separated processes, or is IT considered a part of the business processes? IN5140/Lecture /Slide 51 Dag Sjøberg Cause of waste 4: Wishful thinking Belief in gurus and best practice Wish new technology will lead to improvement in any case Scientific approach with theories, hypotheses, experimentation and data as basis for decision making is ignored Often quick decision based on one alternative Decision can often be deferred and more alternatives considered Testing is not performed properly because hope that the software is good enough without IN5140/Lecture /Slide 52 Dag Sjøberg 26

27 Cause of waste 5: Technical debt A metaphor that describes shortcuts taken during development and maintenance to achieve short-term business benefits but that in the long-term will lead to increased cost of maintenance and further development Lecture 23 October is about technical debt IN5140/Lecture /Slide 53 Dag Sjøberg Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Agile principle 1: Satisfying the customer Flow Visualization Waste Causes of waste Agile principles 2 12 IN5140/Lecture /Slide 54 Dag Sjøberg 27

28 Agile principle 2: Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. IN5140/Lecture /Slide 55 Dag Sjøberg Attitudes towards change Change should not be considered a risk in itself Change is necessary to create better systems Agile teams don t oppose change IN5140/Lecture /Slide 56 Dag Sjøberg 28

29 Lean principle: Just-In-Time (JIT) Make decisions as late as possible, that is, not before they are demanded Many details can be decided after fundamental aspects have been implemented Focus only on details that are necessary to implement the work items in process Waste is avoided: Certain items may turn out not be needed at all Items may be implemented in better ways than if they had been implemented earlier Reduce bottlenecks IN5140/Lecture /Slide 57 Dag Sjøberg Still, must put effort into developing good requirements specifications (see Ch. 5 Larman & Vodde, False dichotomies) The possibility for change should not be used as an excuse for not clarifying fundamental requirements at an early stage Eliciting, analyzing and specifying requirements are as important as coding and testing Many methods and tools are available. They must be learned and adapted to a local context IN5140/Lecture /Slide 58 Dag Sjøberg 29

30 Class exercise Give examples of decisions made too early? What s your attitute towards changing plans? Is it considerd intelligent to change the course according new information being available, or would one be considered a person who doesn t know his own mind (cf. politician) IN5140/Lecture /Slide 59 Dag Sjøberg Agile principle 3: Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. IN5140/Lecture /Slide 60 Dag Sjøberg 30

31 Sprint = iteration on a set of increments of the system Outline planning and architectural design Input: Product backlog list of work Items Assess Select Review Develop Sprint cycle 2-4 weeks Project closure Develop documentation (help functions, user manuals) and summarize what was learned IN5140/Lecture /Slide 61 Dag Sjøberg Waste is reduced by developing smaller increments in frequent iterations May discover at an early point in time that a feature was not needed or useful an increment was not properly designed or tested IN5140/Lecture /Slide 62 Dag Sjøberg 31

32 Agile principle 4: Business people and developers must work together daily throughout the project. IN5140/Lecture /Slide 63 Dag Sjøberg Ensure relevance Continuous testing for bugs isn t sufficient The functionality must be useful IN5140/Lecture /Slide 64 Dag Sjøberg 32

33 How to communicate with the customer? Include Product Owner in the team Participatory design Design by doing instead of extensive documentation Mutual learning IN5140/Lecture /Slide 65 Dag Sjøberg Agile principle 5: Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. IN5140/Lecture /Slide 66 Dag Sjøberg 33

34 Local initiative Management by example Give people confidence to make decisions themselves. Then they become more motivated Lead teams to discover good processes themselves, instead of enforcing processes on them Don t document processes extensively Newcomers should have a gentle introduction to processes, but should have freedom to continuously improve them IN5140/Lecture /Slide 67 Dag Sjøberg Agile principle 6: The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Colocation is important Difficult in distributed teams, use electronic devices Management by walking around IN5140/Lecture /Slide 68 Dag Sjøberg 34

35 Agile principle 7: Working software is the primary measure of progress. Lean: too much documentation and models is waste IN5140/Lecture /Slide 69 Dag Sjøberg Agile principle 8: Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Lean: Taken for granted, basis IN5140/Lecture /Slide 70 Dag Sjøberg 35

36 Agile principle 9: Continuous attention to technical excellence and good design enhances agility. Lean: Continuous focus on quality is central IN5140/Lecture /Slide 71 Dag Sjøberg Agile principle 10: Simplicity the art of maximizing the amount of work not done is essential. (= minimizing the amount of necessary work done) Lean: Reduce waste IN5140/Lecture /Slide 72 Dag Sjøberg 36

37 Just-in-Time By not developing functionality before it s needed, the need may disappear and resources are not wasted IN5140/Lecture /Slide 73 Dag Sjøberg Agile principle 11: The best architectures, requirements, and designs emerge from self-organizing teams. Central in both Norwegian/Nordic work life and Lean IN5140/Lecture /Slide 74 Dag Sjøberg 37

38 Self-organizing and cross-functional teams Cross-functional teams have all the competence needed to carry out all the tasks of an iteration Include people who can test that the code works and domain experts that can test that the code does what it is supposed to do. By being in the same team, means testing can be done continually, minimizing the amount of code waiting to be released. Decisions should be taken where the relevant competence is What if the competence in the team is not good enough? Requirements of self-organized team Everybody should be heard IN5140/Lecture /Slide 75 Dag Sjøberg Agile principle 12: At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. IN5140/Lecture /Slide 76 Dag Sjøberg 38

39 Diagnosis One must find the cause of the problem (root cause analysis) Not just try an arbitrary medicine recommended by best practice or gurus Analysis is necessary, for example, by experimentation IN5140/Lecture /Slide 77 Dag Sjøberg Continuous learning and improvement (Kaizen) focus on details Catastrophes often a result of sequences of small errors Improving processes requires improvements of many (small) elements Consider problems as challenges opportunities for learning, don t blame individuals In retrospectives, problems are identified but often not solved Solutions require time but is an investment (time saved later) IN5140/Lecture /Slide 78 Dag Sjøberg 39

40 01/10/18 IN5140/Lecture /Slide 79 Dag Sjøberg Next lecture: Educational Kanban game by Ketil Jensen, Leverage51 IN5140/Lecture /Slide 80 Dag Sjøberg 40