Copyright Software Engineering Competence Center

Size: px

Start display at page:

Download "Copyright Software Engineering Competence Center"

Robert May
6 years ago
Views:

1 Copyright Software Engineering Competence Center

2 Copyright Software Engineering Competence Center

3 These are mapped categories to the waste categories of manufacturing. An excellent overview of these waste categories in: This topic is very important in Configuration Management, because many traditional CM processes inject many wastes in all these categories. Copyright Software Engineering Competence Center

In simple terms, the four common concepts in these definitions are: 1. Given many copies of the same artifact, you will be able to identify which version each copy is. 2.

4 In simple terms, the four common concepts in these definitions are: 1. Given many copies of the same artifact, you will be able to identify which version each copy is. 2. Ability to induce information about the work products, like their release date, related issues, etc. 3. Make sure that a given set of files constitute a correct version. Also, ability to make sure that a given set of work products fulfill a set of functionality. 4. Verify that released products are packaged correctly, and fulfills requirements. These definitions may not be that understood to practitioners. Why, for example, shall I bother about configuration items specific versions? I would rather make that I m working on a correct branch and that s it! Copyright Software Engineering Competence Center

5 These two are the and most practical definitions of all. It mentions the single a most important objective about configuration management, which is tracking and controlling changes. Copyright Software Engineering Competence Center

6 From another angle, the diagram identifies a property of a strong configuration management environment. This is the ability to trace: Trace releases to its work products. Trace releases to workitems worked in this release. Trace workitems to work products which resulted from them. Trace workitems to their releases. Trace work products to releases. Trace work products to workitems which produced or changed these work products. If I can do this, then I have a strong configuration management environment Copyright Software Engineering Competence Center

7 Copyright Software Engineering Competence Center

8 Following is an explanation of every process increment: Version Control Project configuration items are under version control, and team is trained on basic copyupdate-merge and lock-modify-unlock procedures Baselining Baselining procedure is defined and enforced at points where work product(s) are delivered to an external party Workitem Tracking Issues types are identified, and issues are managed and tracked Traceability Bi-directional traceability of requirements and work products is defined and enforced Release Management Release and release scope is identified; Changes are received, prioritized, and planned; packaging, releasing, and post-release verification procedures are enforced CM Environment Project structure is defined, access rights are enforced, and backup/restore procedure is employed Continuous Integration Builds are automated and integration between team members, and between teams is automated and frequent Copyright Software Engineering Competence Center

9 This is the first and most basic technical practice in software development. However, you may still find so many teams doing version control in an unproductive way. Copyright Software Engineering Competence Center

10 Identifying configuration items is an important task in software projects. It is not enough to put your code on svn. There may be other work products that need to be versioned. For example, consider a vast product developed by several teams. At first, an architecture team agrees on the architecture, develop component diagram, and identify components interfaces. Then, teams start working, and after a while, the product get. In this case, the architecture document is a configuration item, and need to be baselined. If not, there is a great risk that teams fail to integrate their work successfully. Copyright Software Engineering Competence Center

11 Copyright Software Engineering Competence Center

12 This is the first and most basic technical practice in software development. However, you may still find so many teams doing version control in an unproductive way. Copyright Software Engineering Competence Center

13 The issue here is what would be the nature of a change management process? In Agile, change management process is the usual development process, nearly no overhead to mention. Many companies employ a sophisticated change management procedures which incur a lot of cost to introduce a single change request. This effectively keeps customers afraid from changing anything, and fires back at time of deployment. Copyright Software Engineering Competence Center

14 Again, take care of the baselining procedure. It is tempting to put all kinds of overhead in this procedure. Copyright Software Engineering Competence Center

15 Copyright Software Engineering Competence Center

16 Workitem tracking is a central topic in Configuration Management. However, agile team usually overlooks it. Most of the teams I have worked with (even teams thinking they are Agile), do not have a clear definition of what workitem types they should track. Even if they have a basic setup of Scrum workitems (user story, task, bug, test case), they do not have an understanding of what state information they should collect, what kind of relations to be maintained between workitems, and what are the optimal workflows for every workitem type. One sign of a mature agile team is that they start creating new workitem types which suites their own environment, and store specific state information about every type. They also know what to do with these workitems. They not only use it to track their everyday development activity, but they also use it to generate release reports, document customer correspondence, and advanced reporting. Copyright Software Engineering Competence Center

17 Workitems are things that are carried out throughout the project. In Project Management terms, these are the Work Breakdown Structure (WBS) of the project. Many teams limit themselves to predefined types in management frameworks like Scrum, RUP, etc. Or even to predefined types in tracking tools. Workitems need to be well thought off with special attention to two things: Existing management system, and Hidden Factory. Existing Management System/Framework: Organizations give different meanings to terms like: Requirement, Enhancement, Change Request, Defect, Support Request, Trouble Incident, etc. It is very important to identify what is meant by the name, and when it should be used. In general, it is advisable to re-use the same terms already used in the organization, but it is necessary to write down what is meant by every term, and when it should be used. Hidden factory: Hidden Factory can be defined as the set of tasks that are done to amount for the process inefficiency. For example, rework due to integration bugs, technical debt accumulated due to very little time spent on unit testing, etc. These tasks are usually not tracked as workitems, or are never tracked. Usually, a good practice that Agile teams do is that these workitems are done as part of several intermediate stabilization iterations, or at the end of every release as Release Iteration. However, if such work get accumulated, tracking is usually overlooked in order not to let others discover the process/team inefficiency. The golden rule is: Hidden factory is bad. Try to eliminate it. If you can t, track it as part of your process Another factor that may help identify workitems is value-stream mapping. This is a technique used in Lean manufacturing to track the progress and flow of materials and information. In Software development, value stream mapping help identify activities and intermediate work-products which result from each activity. Kanban boards are an example of a simple value stream, in which materials (requirements) and products (software) are mapped and tracked. Copyright Software Engineering Competence Center

Workitem relationships are rules defined as part of the definition of the workitem system. Usually, they are optional, but in other times they are mandatory.

18 Workitem relationships are rules defined as part of the definition of the workitem system. Usually, they are optional, but in other times they are mandatory. In some teams, workitem relationship rules are not defined, although all kinds of relationships is open to create (using a tool, may be). This is an example of a process that may never be improved, because there is no process to start with. It is important to note that having a process definition is half way towards process improvement. There are many examples of critical situations where we needed to trace some workitem to another and we couldn t, due to the loose rules defined in workitem relationships. Copyright Software Engineering Competence Center

19 Automating workitem tracking systems is a controversial issue. Many teams do workitem tracking on a Scrum or Kanban Boards, and that s it. Many others use simple excel sheets, one sheet per iteration or release. This level of automation is very little to achieve lean implementation of configuration management. For example, we need to report the status of what has been accomplished in this release. If we do not systematically track workitems on a tracking tool, it will be an overhead activity to generate this report from row paper or simple excel sheets. Another point is how backup will be done, who is responsible for it, and how restore will take place? Are Workitems configuration items? Usually yes. Because we need to track the history of changes done to this workitem. Can they be part of a baseline? How? Sometimes we need to include a use-case survey report as part of the baseline, or attach a list of user stories delivered for every release. In this case, they are baselined. Copyright Software Engineering Competence Center

20 Traceability is one of the major and key process increments of configuration management. As will be explained later, good traceability policy and implementation empowers the team with techniques to do impact analysis, root cause analysis, and status accounting. These tools are key tools in improvement. Nevertheless, still a lot of voices are talking about traceability as a real challenge, and a major problem area in the industry. It may induce costs and overheads which grows exponentially with the size and complexity of the software system 1. Also, despite many advances, RT remains a widely reported problem area by industry 2 This is why most of the teams (regardless team or project size) I have worked with do not pay attention to implementing traceability. And, the few teams which employs the so called Traceability Matrix are complaining that it is pure overhead, and they are only doing it for compliance purposes. 1 Kannenberg.pdf 2 F8A63C958BA3EF3176E8AFD0B48687DA/643F156E ECF56449AA/749/1/2.2_rtprob.pdf Copyright Software Engineering Competence Center

21 I found teams who implement traceability for compliance purposes mix traceability with traceability matrix. Traceability matrix is one way of implementing traceability. Ok, what is traceability? Rather than introducing a formal definition, it is easier to look at the network and deduce what traceability is. Requirements entities can trace to each other (white), and trace to other information (green), and finally trace to physical artifacts (gray). Copyright Software Engineering Competence Center

22 Forward traceability enables the team to study the impact of a change and assess its costs and risks. In this scenario above, the customer has requested a change on an already implemented user story. To assess the change, the team revisits the written code, the impacted design and products, etc. Copyright Software Engineering Competence Center

23 If there is a defect reported by a customer, and we want to know its root cause. As in the scenario above, the defected code is traced back to the reasons of change, which is found to be a change request by a customer requested sometime ago. This is also called Backward traceability. Copyright Software Engineering Competence Center

24 Any kind of reporting in software projects makes use of traceability one way or another. Copyright Software Engineering Competence Center

25 These are the set of objectives which need to be fulfilled: 1. How release and release scope are identified 2. What is the procedure for receiving changes, then prioritizing and planning them? 3. Why and how baselines take place, and what is included in baselines? 4. How does the team package their work, and what is the procedure for releasing and verifying releases at customer environment? Some Agile teams overlook many details of the above, and this may cause problems throughout the project. The good news is that Agile methods have done excellent job defining release management procedures. In the next slides, I will try to identify some Agile practices that does the above requirements very well. Copyright Software Engineering Competence Center

26 These are some rules defined in Scrum, and governs how release management is done. First, the customer can propose changes anytime, during review meetings, while estimating, during informal reviews inside iterations, etc. What is important is that these changes are not carried out on spot, they are kept in a Backlog of workitems. The Customer and the Product owner work together continuously to refine the details of the backlog items, and activity called Backlog Grooming, in which: The backlog items are detailed Big requirements are broken down Business priorities are revised Backlog Grooming is a continuous activity that continues to take place throughout the project. Requirements that reached a good stability level are estimated. Then, next iteration s scope is identified based on business priority, technical risk, and cost. This effectively fulfills the necessary impact analysis for the flow of changes requested by the customer. Still, there is a very important rule in the next slide Copyright Software Engineering Competence Center

27 This rule is a golden rule which governs the change management process in Scrum, and makes change control possible. Some agile teams do not abide by this rule, and it results in the following symptoms: - Team receives inconsistent requirements or even contradictory requirements, which makes the team override several old requirements - Velocity cannot be calculated, and iteration length is not kept constant - Team start to be less productive, and customers start to complain The next slide outlines an end-to-end change management procedure in a typical Scrum Agile team. Copyright Software Engineering Competence Center

This is a typical change management process in Agile. This is a description of the process: 1. Requirements and other request types are gathered into a backlog as backlog items 2.

28 This is a typical change management process in Agile. This is a description of the process: 1. Requirements and other request types are gathered into a backlog as backlog items 2. Team (including product owner and/or customer representatives) plans the next release. 3. Development team plans the next iteration, and start development. 4. After iteration ends, the team set for a demo and iteration review. 5. Typically, changes are requested, which are gathered in the backlog. 6. Impact is assessed and may result in changing the scope of the release, which in turn may result in changing the contract. 7. Re-iterate In the meanwhile, changes may occur due to other business circumstances other than usual review in the iteration review meetings. These are referred to as Business Changes (drawn as an accept event action). What is interesting about this process is that changes do not need a special procedure to be developed. In other words, the development process is the same as the change management process. Copyright Software Engineering Competence Center

29 This is one of the interesting Agile practices: Development Process is the Change Management Process What is interesting about it is that it removes so many wastes due to a separate change management process. In traditional processes, change management is done in a separate path (or even separate process) rather than the development path, and this introduces the following wastes: 1. Task switching between the development and change management tasks. 2. If two team are involved, late integration between the teams may introduce defects. 3. If teams who apply changes are different from the development team, learning is required, which is another type of waste. If they do apply changes without proper learning, they may introduce defects. 4. Extra management effort to plan, estimate, track two separate paths of development. In large teams, these problems are magnified, and may cause the development process to halt altogether, and all the team only work on high priority changes only! Copyright Software Engineering Competence Center

30 For teams larger than 10, it is advisable to split the team, and let every sub-team take a specific responsibility. splitting may be done on business feature/function base, or on architectural module base. If you note, the roles above is slightly changed. There is a Chief PO who takes care of the business along with the team PO s. Also, there are multiple Scum masters who are representing their teams in product level planning and analysis. Several notes about this approach: 1. There are always integration risks resulting from the fact that multiple teams working separately for a time period, then come together and integrate. In general, it is advisable to integrate frequently, but not that much. This means that every team is responsible to integrate his work on a mainline and keep all other work unchanged. This will come later in Release Branching Patterns. 2. Product level impact analysis activities are taken up one level, to the group of Scrum Masters working on the product level. 3. Change management is typically the same. However, on the product level, it takes place several iterations (every release). In this case, the product should be planned to have several internal releases before any public release is available. Copyright Software Engineering Competence Center

31 This is a typical packaging and releasing procedure. These procedures are very risky and if not managed well may result in project failure and a lot of overhead costs. Several important waste: - Environments are not defined, which may inject defects and incur relearning costs every time a deployment is done. - Improper baselining take place, or very basic code baseline. - Integration defects due to very late or non-frequent integration - System integration defects due to un-defined system interfaces - Most important one: Defects injected due to human factor. Any manual step in the process may inject defect. The key solutions in lean packaging and releasing are: 1. Automate as much as you can of this procedure to prevent defects due to manual operation 2. Frequent integration (every iteration at least). This is related to the Done-Done definition, an interesting practice in Agile. In large and risky projects, a Done- Done definition which stops at the development (or even testing) environment is a poor definition, because it concentrates integration risk at the end of the release. Rather, a typical mitigation for integration risks is to integrate frequently every iteration. Copyright Software Engineering Competence Center

32 For small teams, CM environment can be set up in one or two hours. However, for larger projects with multi-teams, CM environment have other dimensions of complexity, and need to be well thought off, and unified across all teams. These are some complexities related to large teams: - Team members may move across teams and will need to re-learn the CM environment upon every move. - Code level integration depends on a unified view of the project structure on the CM tool. - Release management may not be possible except if proper release level branching & merging techniques are employed. Otherwise, incorrect releases may take place and costly post-deployment defects may incur so much extra costs. Copyright Software Engineering Competence Center

33 Copyright Software Engineering Competence Center

34 A unified project structure is very important. In a multi-team project, all team should work in harmony. This is just an example of a project structure. These are some notes: - The project has three mainstream called Trunc - There are other active streams called Branches every branch is an active code stream that may be updated anytime a new change request is approved. In this case, the responsible team checks out this branch and start working on it - The last folder is the Tags, which keeps baselined streams - For work products which may not be part of any baseline (like meeting minutes, and periodic reports), it may be kept in a special Archive folder Copyright Software Engineering Competence Center

35 Copyright Software Engineering Competence Center

36 Backup & restore procedure is an indicator of team discipline. A team with no or inactive backup plan is a team which indicates little discipline and is introducing a great risk in the project. Copyright Software Engineering Competence Center

37 The intent of this section is to highlight the topic of Release branching & merging. The patterns presented are just a sample of so many others. Release level branching & merging is a different from individual development branching & merging practices. The former deals with the whole release, which is a responsibility of one whole team. The latter deals with an everyday development activity. Factors that may determine which pattern to follow are: - Number of development teams working in parallel - Number of customers and amount of customization for every customer - Cost or risk of Integration - Required release frequency Copyright Software Engineering Competence Center

38 Copyright Software Engineering Competence Center

39 Copyright Software Engineering Competence Center

40 Copyright Software Engineering Competence Center

41 Copyright Software Engineering Competence Center

42 Copyright Software Engineering Competence Center

43 Due to the major risks related to integration. Continuous integration is added as a standalone process increment for Configuration Management. Copyright Software Engineering Competence Center

44 This is the de-facto model for integration between teams. This model has so many problems and cause so many delays. The reasons why this model doesn t work is mentioned in the next slide. Copyright Software Engineering Competence Center

45 Copyright Software Engineering Competence Center

46 Copyright Software Engineering Competence Center

47 Copyright Software Engineering Competence Center

48 Copyright Software Engineering Competence Center

Software Engineering Part 2

CS 0901341 Software Engineering Part 2 In this part, we look at 2.1 Software Process 2.2 Software Process Models 2.3 Tools and Techniques for Processing Modelling As we saw in the previous part, the concept