INF5181: Process Improvement and Agile Methods in Systems Development

Size: px

Start display at page:

Download "INF5181: Process Improvement and Agile Methods in Systems Development"

Robyn Booker
6 years ago
Views:

1 INF5181: Process Improvement and Agile Methods in Systems Development Lecture 26 September 2017: Lean and Agile Software Engineering Yngve Lindsjørn INF5181->Lean and Agile Software Engineering 1

2 Relationship between lecture and book This lecture focuses on the stuff in Chapters 2, 3, 4 (indirectly reduce cycle time) and 5 (indirectly) and 6 of the Larman & Vodde book. Emphasizes the relationship between Lean and Agile (more than the book does) The relationship between process improvement, measurements, empirically-based/evidence-based decision making, Lean and Agile is clear: -> It s about working smarter Believe lectures of INF5181 are better structured than any text book INF5181->Lean and Agile Software Engineering 2

3 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Agile principles 3-12 INF5181->Lean and Agile Software Engineering 3

4 Systems thinking Systems thinking is the process of understanding how things, regarded as systems, influence one another within a whole. In nature, systems thinking examples include ecosystems in which various elements such as air, water, movement, plants, and animals work together to survive or perish. In organizations, systems consist of people, structures, and processes that work together to make an organization "healthy" or "unhealthy". Systems thinking has roots in the General System Theory that was advanced by Ludwig von Bertalanffy in the 1940s and furthered by Ross Ashby in the 1950s. [Wikipedia 2014] INF5181->Lean and Agile Software Engineering 4

5 Systems thinking is about understanding the effect of actions on the total system Global optimization As opposed to sub-optimization, which is local optimization System dynamics Consequences/Feedback loops Root cause analysis Usually several direct and indirect causes INF5181->Lean and Agile Software Engineering 5

6 Technique for root cause analysis Ishikawa diagram* indicates relationship between (un)wanted effects (Problem/Solution) and their possible causes Environment Humans Software Problem/Solution Methods Procedures Hardware *After the Japanese professor Kaoru Ishikawa also called fishbone diagram INF5181->Lean and Agile Software Engineering 6

7 Observed problem: Ineffective ScrumMasters *From Scaling Lean and Agile Development: p 31 INF5181->Lean and Agile Software Engineering 7

8 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Kahoot quiz INF5181->Lean and Agile Software Engineering 8

9 Purpose of Lean All parts of the organization should optimize their processes to achieve a common goal: To create value for the customer Value = what the customer wants and is willing to pay for INF5181->Lean and Agile Software Engineering 9

10 Overall Lean principles Customer value Satisfying the customer Flow Visualization Avoiding waste Supporting change Magic triangle INF5181->Lean and Agile Software Engineering 10

11 Lean production The Japanese school, primarily Toyota If failure discovered, stop the assembly line and find the reason for the failure instead of only collect and fix the failures in bulks (root cause analysis and root cause fix) Continuous learning and improvement (Kaizen) Just-in-time (JIT) principle: don t produce anything before somebody demands it Tempo in production determined by pull from customer or next element in the production chain rather than push internally to produce as much as possible Removal of temporary storage Component-based production (same chassis, bumper, etc. on different car models). Production geared towards the customer, components can be outsourced to third-party vendors Result: Toyota fewest failures and fastest production The most productive factories spent least resources on management and administration ( lean management )

12 Lean stems from Japanese car industry, but Now widespread in all kinds of knowledge-intensive industry and administration Large companies Hospitals Governments Universities Etc. INF5181->Lean and Agile Software Engineering 12

13 Similarities with the Norwegian/Nordic model Self governed (autonomous) team Learning, redundancy/job-rotation Work environment Quality of life Co-operation among management, trade unions and government Volvo, Hydro and many more B. Gustavsen, T.U. Quale, B.A. Sørensen, M. Midtbø og P.H. Engelstad. Innovasjonssamarbeid mellom bedrifter og forskning den norske modellen. Gyldendal 2010 INF5181->Lean and Agile Software Engineering 13

14 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Agile principles 3-12 INF5181->Lean and Agile Software Engineering 14

15 INF5181->Lean and Agile Software Engineering 15

16 How the manifesto has been interpreted and often practised [Janes & Succi 2012]

17 A balancing act [Bell & Orzen 2011]

18 Agile Manifesto 12 principles* 1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. 2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. 3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. 4. Business people and developers must work together daily throughout the project. 5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. 6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. 7. Working software is the primary measure of progress. 8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. 9. Continuous attention to technical excellence and good design enhances agility. 10. Simplicity the art of maximizing the amount of work not done is essential. 11. The best architectures, requirements, and designs emerge from self-organizing teams. 12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. * INF5181->Lean and Agile Software Engineering 18

19 Scrum meetings and the 12 agile principles Exercise on the exam (2015) a. Give a brief description of the daily scrum meeting, the sprint planning meeting and the review meeting in large scale scrum (suppose you have a project with more than 10 scrum teams). b. Give a brief description of which of the 12 agile principles (see appendix) that supports these meetings and in what way they do it. * INF5181->Lean and Agile Software Engineering 19

20 What are the relationships between the 12 agile principles and Lean? INF5181->Lean and Agile Software Engineering 20

21 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Agile principles 3-12 INF5181->Lean and Agile Software Engineering 21

22 Agile principle 1: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. INF5181->Lean and Agile Software Engineering 22

23 Who is the customer? One who pays for the system, uses the system and/or gets value from the system Consider the whole value chain, including customers of the customer Customer may be internal or external (think total system) INF5181->Lean and Agile Software Engineering 23

24 A software team must relate to many stakeholders: What are they expectations? Customers: Do they prioritize everything, perfect and immediate delivery, or are they more realistic? Management: Consider overall goals, expect no surprises? Developers: Expect to build with top technical quality of technical, fancy system? Maintainers: Expect few bugs, understandable and well documented code? Other stakeholders (law makers, other systems) INF5181->Lean and Agile Software Engineering 24

25 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Kahoot quiz INF5181->Lean and Agile Software Engineering 25

26 Flow Fundamental in Lean, less focus in Agile INF5181->Lean and Agile Software Engineering 26

27 Total value stream systems thinking What is the customer s purpose with the system? How to ensure flow of a customer s request or product through the whole value stream? Value stream map of the organization to identify waste possibilities for optimizing the processes INF5181->Lean and Agile Software Engineering 27

28 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Agile principles 3-12 INF5181->Lean and Agile Software Engineering 28

29 Visualization INF5181->Lean and Agile Software Engineering 29

Obeya (war room) A form of project management, where all those involved in planning (of a project) gather in one room Visual charts and graphs depicting:

30 Obeya (war room) A form of project management, where all those involved in planning (of a project) gather in one room Visual charts and graphs depicting: Vision/objectives Plan with milestones and progress Task board Metrics etc Reduces need for communication INF5181->Lean and Agile Software Engineering 30

31 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Agile principles 2-12 INF5181->Lean and Agile Software Engineering 31

32 To help ensure flow, reduce waste INF5181->Lean and Agile Software Engineering 32

33 Waste definition Everything that requires resources time work effort space equipment money and that does not give value to the customer INF5181->Lean and Agile Software Engineering 33

34 Waste examples 1. Overproduction 2. Wait 3. Transport. 4. Overprocessing 5. Queue/ inventory 6. Rework 7. Motion/ Task switching 8. Skills 9. Information scatter/loss 10. Wishful thinking

35 [Larman & Vodde, p.60-61]

36 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Agile principles 3-12 INF5181->Lean and Agile Software Engineering 36

37 Five important causes of waste in software development Complexity Scale instead of flow Separating decision making from work Wishful thinking Technical debt [Poppendieck & Poppendieck 2010] INF5181->Lean and Agile Software Engineering 37

38 Be a bit sceptical to data from Standish

40 Unnecessary complexity Waste category Examples Over processing Too complex business processes are automated Software development processes and best practices that give overhead Too much design too early Extra features Many features of the systems are not used Gold plating system design and performance Documentation is often not used Identify functionality that is not strongly needed Don t implement at all or defer to JIT (Just-in-time) Most code is rarely or never executed Don t measure individuals performance in LOC or Function Points (the code should be minimal) Improve work and business processes before they are automated INF5181->Lean and Agile Software Engineering 40

41 Cause of waste 2: Scale instead of flow Traditional thinking that one needs to scale up to increase profit Efficient flow may be more important Challenges: At present, there may be more focus on individual tasks than the whole value stream Too many tasks in parallel (WIP) INF5181->Lean and Agile Software Engineering 41

42 Task switching and waiting Waste category Examples Task switching People are frequently rushed out to seemingly more urgent things Wait Work waiting to be reviewed, approved and forwarded Delay between coding and testing Waiting for specialists who are assigned elsewhere Waiting for tools, licenses.. INF5181->Lean and Agile Software Engineering 42

43 Are you most efficient when you do many or few things in parallel? INF5181->Lean and Agile Software Engineering 43

44 Cost of context switching / multitasking Figure by Mike Cohn based on Clark and Wheelwright (1993)

45 Cause of waste 3: Separating decision making from work Managers must understand the work, that is, be able to ask relevant questions Are business processes and IT considered as separated processes, or is IT considered a part of the business processes? INF5181->Lean and Agile Software Engineering 45

46 Cause of waste 4: Wishful thinking Belief in gurus and best practice Wish new technology will lead to improvement in any case Scientific approach with theories, hypotheses, experimentation and data as basis for decision making is ignored Often quick decision based on one alternative Decision can often be deferred and more alternatives considered Testing is not performed properly because hope that the software is good enough without INF5181->Lean and Agile Software Engineering 46

Cause of waste 5: Technical debt A metaphor that describes shortcuts taken during development and maintenance to achieve short-term business

47 Cause of waste 5: Technical debt A metaphor that describes shortcuts taken during development and maintenance to achieve short-term business benefits but that in the long-term will lead to increased cost of maintenance and further development INF5181->Lean and Agile Software Engineering 47

48 Technical debt = Everything that makes it difficult to modify code Since code always must be modified, we have to pay back the debt. If not, we are bankrupt. Write clean code, that is, concise, simple with few dependencies New functionality leads to larger code base and increased complexity; refactoring reduces the debt INF5181->Lean and Agile Software Engineering 48

49 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Agile principles 3-12 INF5181->Lean and Agile Software Engineering 49

50 Agile principle 2: Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. INF5181->Lean and Agile Software Engineering 50

51 Attitudes towards change Change should not be considered a risk in itself Change is necessary to create better systems Agile teams don t oppose change INF5181->Lean and Agile Software Engineering 51

52 Lean principle: Just-In-Time (JIT) Make decisions as late as possible, that is, not before they are demanded Many details can be decided after fundamental aspects have been implemented Focus on only details that are necessary to implement the work items in process Waste is avoided: Certain items may turn out not be needed at all Items may be implemented in better ways than if they had been implemented earlier Reduce bottlenecks INF5181->Lean and Agile Software Engineering 52

53 Structure Systems thinking (Ch. 2 Larman & Vodde) Overview of Lean and Agile Lean versus Agile Satisfying the customer Flow Visualization Waste Causes of waste Supporting change Agile principles 3-12 INF5181->Lean and Agile Software Engineering 53

54 Agile principle 3: Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. INF5181->Lean and Agile Software Engineering 54

55 Increments and iterations in systems development An increment is an addition to the functionality an aspect of the system An iteration is a cycle in the development an aspect of the process Typically, a new increment is developed through a new iteration A new iteration may also improve the quality of the same functionality, that is, no new increment is being developed INF5181->Lean and Agile Software Engineering 55

56 Waste that can be reduced by developing smaller increments in frequent iterations Waste category Examples Defects Customer/user requirements and expectations are not understood Customer/user needs are assumed, but not verified Errors resulting from insufficient analysis and reflection Bad design of software Partially done work Projects and programs that are not really considered, are kept on the backlog Software requirements are specified long before coding Unsynchronized code, untested code, undocumented code, undeployed code. INF5181->Lean and Agile Software Engineering 56

57 Agile principle 4: Business people and developers must work together daily throughout the project. INF5181->Lean and Agile Software Engineering 57

58 Ensure relevance Continuous testing for bugs isn t sufficient The functionality must be useful INF5181->Lean and Agile Software Engineering 58

59 How to communicate with the customer? Include Product Owner in the team Participatory design Design by doing instead of extensive documentation Mutual learning Face-to-face communication most effective but may be difficult in distributed projects (instead, video conferences) INF5181->Lean and Agile Software Engineering 59

60 Agile principle 5: Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. INF5181->Lean and Agile Software Engineering 60

61 Local initiative Management by example Give people confidence to make decisions themselves. Then they become more motivated Lead teams to discover good processes themselves, instead of enforcing processes on them Don t document processes extensively Newcomers should have a gentle introduction to processes, but should have freedom to continuously improve them INF5181->Lean and Agile Software Engineering 61

62 Agile principle 6: The most efficient and effective method of conveying information to and within a development team* is face-to-face conversation. - Colocation is important - Electronic devices in distributed teams *Cross-functional, Larman & Vodde call it: feature team INF5181->Lean and Agile Software Engineering 62

63 Agile principle 7: Working software is the primary measure of progress. Lean: too much documentation and models is waste INF5181->Lean and Agile Software Engineering 63

64 Agile principle 8: Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Lean: Taken for granted, basis INF5181->Lean and Agile Software Engineering 64

65 Agile principle 9: Continuous attention to technical excellence and good design enhances agility. Lean: Continuous focus on quality is central INF5181->Lean and Agile Software Engineering 65

66 Agile principle 10: Simplicity the art of maximizing the amount of work not done is essential. (= minimizing the amount of necessary work done) Lean: Reduce waste, e.g., by JIT INF5181->Lean and Agile Software Engineering 66

67 Just-in-Time By not developing functionality before it s needed, the need may disappear and resources are not wasted INF5181->Lean and Agile Software Engineering 67

68 Agile principle 11: The best architectures, requirements, and designs emerge from self-organizing teams. Central in both Norwegian/Nordic work life and Lean INF5181->Lean and Agile Software Engineering 68

69 Self-organizing and cross-functional teams Cross-functional teams have all the competence needed to carry out all the tasks of an iteration Include people who can test that the code works and domain experts that can test that the code does what it is supposed to do. By being in the same team, means testing can be done continually, minimizing the amount of code waiting to be released. Decisions should be taken where the relevant competence is What if the competence in the team is not good enough? Requirements of self-organized team Everybody should be heard INF5181->Lean and Agile Software Engineering 69

70 Agile principle 12: At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. INF5181->Lean and Agile Software Engineering 70

71 Diagnosis One must find the cause of the problem (root cause analysis) Not just try an arbitrary medicine recommended by best practice or gurus Analysis is necessary, for example, by experimentation INF5181->Lean and Agile Software Engineering 71

72 Continuous learning and improvement (Kaizen) focus on details Catastrophes often a result of sequences of small errors Improving processes requires improvements of many (small) elements Consider problems as challenges opportunities for learning, don t blame individuals In retrospectives, problems are identified but often not solved Solutions require time but is an investment (time saved later) INF5181->Lean and Agile Software Engineering 72

73 INF5181->Lean and Agile Software Engineering 73

74 Timeboxing versus task-boxing Scrum has sprints (iterations) of 2-4 weeks with defined start and end activities Not always easy to divide the tasks or features of the systems to fit into such time intervals, e.g., maintenance and support tasks What about instead work on a prioritized set of tasks until they re finished? (Kanban) INF5181->Lean and Agile Software Engineering 74

IN5140 Smart processes and agile methods in software engineering

IN5140 Smart processes and agile methods in software engineering Lecture 2 October 2018: Lean and Agile Software Engineering Dag Sjøberg E-mail: dagsj@ifi.uio.no IN5140/Lecture 2018.10.02/Slide 1 Dag Sjøberg