Agile Software Production?

Transcription

1 Agile Software Production? Andrew Patterson Electric Cloud Europe

2 Agenda Its not (just) about Agile Development The real issues in the Software Development Lifecycle What are they? What is important? Software Production : Tool Options Buy or Build? How do you decide?

3 The Ideal? Source Tree Build Test Deploy Agile Build, Test and Deployment High Degree of Automation

4 LifeCycle Tool Investment CODE BUILD TEST DEPLOY SCM $2 B Test Creation $1.5 B Software Creation Dev Tools Multi-Billion Software Production? Slide 4

5 SPM Problems Resource intensive to maintain Slower time to market Slow Not accessible to engineers Not Repeatable Brittle Lower quality Lack of compliance Roadblock to modern development techniques Agile Global teams Virtual environments Lost Productivity Not Standard Slide 5

6 Software Production Checklist Topic Build times > 30 minutes Slow, manual steps from SCM->Build->Test Need Continuous Integration Have problems managing and maintaining build scripts / single person has knowledge Need to build on multiple platforms Developers lose time waiting for overnight builds, test results Project releases would be earlier if multiple platform build and test run in parallel Need visibility/metrics on build and test progress Have distributed teams sharing common build and test processes Response Comment Multiplies up with variants, localisation Time consuming Help with time-tomarket, code quality High-risk Can be a serial, slow process Developers switch projects while waiting Don t know what can be parallelised Manual work today Increasingly common

7 Production Time Example 1 SmartPhone Ground-Up Build time 37 hours 4 Hardware Platforms 22 Localisation builds for target languages 135 days total Incremental builds and links can lead to dependency issues One file change / bug fix and start-over

8 Project Trade-off. You may only have three of these : Good Fast Cheap Done (Quality) (Time to Market) (Cost Effective) (Project Completed) But, You can optimise the mix

9 What is the Enterprise Requirement?

10 Linking Distributed Teams SW DEVELOPERS Copenhagen ENGINEERING MGR Boston BUILD TEAM San Francisco SW DEVELOPERS Bulgaria OUTSOURCED QA Bangalore Specific Access and Permissions Based on Role Anywhere in the World Slide 10

11 Managing Resources ENGINEERING MGR SW DEVELOPERS VMware Lab Manager VMware Lab Manager BUILD TEAM Dynamic Provisioning VMware Lab Manager SW DEVELOPERS OUTSOURCED QA Build Servers Test Servers Production Servers Virtual Servers Slide 11

12 Integrate Tools and Processes SW DEVELOPERS Deployment Tools Unit SCM Tests Tools Build Tools SW DEVELOPERS ENGINEERING MGR BUILD TEAM OUTSOURCED QA Build Tools Unit Tests SCM Tools Deployment Tools Build Servers Test Servers Production Servers Virtual Servers

13 Tie it all Together SW DEVELOPERS SW DEVELOPERS ENGINEERING MGR BUILD TEAM OUTSOURCED QA Build Tools Unit Tests SCM Tools Deployment Tools Build Servers Test Servers Production Servers Virtual Servers Slide 13

14 Blue Sky Solution Container for processes and results Process to run automatically Command or script Project Procedure Checkin Code Set Up Java Build Compile Unit Test Report When to run a procedure Schedule/Trigger Job Job Job Job Pool Linux server Linux server Linux server Windows server Test harness Records results of running a procedure Resources Slide 14

15 Managing Build and Test Data Fixed Product-specific Job-specific server name password repository location directory branch targets Configuration Inputs Cross-Build Info last green build counter for build ids Job process id resource to use Working Data Job Results success/ failure error count warning message QA status promotion level problem notes Post-Completion Info

16 Visibility: Management Reports Slide 16

17 Methodology : Continuous Integration and Pre-Flight Builds

18 Continuous Integration Check-In SCM System QA Build Success Automated Workflow Developer runs local build and unit tests Developer checks tested code into SCM system Integration build at frequent intervals or upon check-in Production Servers Integration Build Build Failure

19 Frequent Problem: Continuously Broken Builds Wait X Check-In Back Out X Wait SCM System Automated Workflow Production Servers Integration Build Developer builds/tests on local system, checks in code Integration build started, breaks ( it worked on my machine ) Team impacted while check-in is backed out or build fixed Build Failure

20 Solution: Pre-Flight Builds and Tests Automated Workflow Success SCM System Developers build and test across all targets/platforms Ensures successful integration build Developers can check in changes with confidence Developers Test Servers Production Servers Broken builds less likely to affect the entire team Build or Test Failure

21 What Actually Happens Get Source Build Windows Install Test Build Linux Install Test Wait For Results System Tests Report/ Notify Release

22

23 The MVP Who are the engineering team MVPs? Managers? Developers? QA? The MVP is the build manager supporting the script Why?

24 Who Else? The build manager actually builds the product that s shipped Sure, developers write the features and bugs Sure, QA tests that the product works Sure, managers do something valuable But the build manager that guy actually built the thing your company ships that guy probably stayed up until 3am to make it happen So why does that guy get no decent tools?

25 The Rise and Rise of the MVP Build Manager In the old days.. TBM didn t exist, being TBM was a role taken on by a developer Developers hated to be the build guy for the day The software grew; the build grew TBM role is formalized, but developers still looked down on him MVP Today The Manager invites TBM to join his weekly staff meeting; TBM matters now Developers, QA and Management harass him about broken builds

26 Quick Win : Accelerating the Software Production Process

27 A typical Software Production process 1 Hour and Growing 10 min 6 min 20 min BUILD Cpp UNIT Doxygen C-Depend Coverage Static Analysis Lint Package Deploy Slide 27

28 Acceleration 90 Minute Process BUILD Cpp UNIT Doxygen C-Depend Coverage Static Analysis Lint Package Deploy Slide 28

29 Acceleration 90 Minute Process BUILD Cpp UNIT Doxygen C-Depend Coverage Static Analysis Lint Package Deploy 15 min 6 min 10 min Slide 29

30 Acceleration BUILD Coarse Grain Parallelism Cpp UNIT Deploy Doxygen Coverage Package Deploy 3x Gain C-Depend Static Analysis Lint 31 Minutes 15 min 6 min 10 min Slide 30

31 Builds Grow BUILD Cpp UNIT Deploy Coverage Package Doxygen Deploy C-Depend Static Analysis Lint 15 min 6 min 10 min Slide 31

32 Builds Grow BUILD Cpp UNIT Deploy Coverage Package Doxygen Deploy C-Depend Static Analysis Lint 3 Hours 15 min 6 min 10 min Slide 32

33 Builds Grow BUILD Doxygen Cpp UNIT Coverage Package Deploy Deploy C-Depend Static Analysis Lint 3 Hours 15 min 6 min 10 min Slide 33

34 The Problem: Dependencies Dependencies (Hidden and/or Expressed) BUILD 3 Hours Doxygen C-Depend Static Analysis Lint Cpp UNIT Coverage Package Deploy Deploy Distributing the Build can solve this problem 15 min 6 min 10 min Slide 34

35 The Problem: Dependencies Dependencies (Hidden and/or Expressed) BUILD 3 Hours Doxygen C-Depend Static Analysis Lint Cpp UNIT Coverage Package Deploy Deploy 15 min 6 min 10 min Slide 35

36 Fine Grain Parallelism on Builds BUILD BUILD Doxygen Cpp UNIT Coverage Doxygen Cpp UNIT Coverage Package Deploy Package Deploy Deploy Deploy C-Depend Static Analysis Lint C-Depend Static Analysis Lint 3 Hours 15 min 15 min 6 min 10 min Slide 36

37 Getting There: DIY Fast builds Buy lots of SMP hardware and try out GNU Make parallelism or manually parallelize the build. Scheduled builds Use cron for that On demand builds Build an intranet page, integrate it yourself with the current build and source code management system Stimuli builds Build ad-hoc script attached to source code management system

38 DIY You could try that Before you do think about How much time do you have to write all that code? How you are going to manage 200+ builds per day? What build time acceleration is needed? How are you going to manage hardware failure in your build system? Who will manage and maintain the solution What is the long-term cost and risk

39 Build Your Own? Seems pretty straightforward Investment Automated build and test for one build tool, one machine Scheduled jobs Features Trigger builds after check-ins

40 Build Your Own? Just a few finishing touches Investment Automated build and test for one build tool, one machine Trigger builds after check-ins Scheduled jobs Database of build results Web-based access to results Features Notifications via /rss CruiseControl gives you this for Java/Ant

41 Build Your Own? Multiple servers, concurrency Investment Notifications via /rss Resource pooling, load balancing Automated build and test for one build tool, one machine Run multiple builds simultaneously Scheduled jobs Trigger builds after check-ins Database of build results Web-based access to results Features Multiple jobs run Multiple servers, remote invocation on one resource simultaneously Run job steps in parallel Single job can use multiple resources Resource selection criteria

42 Build Your Own? Better monitoring and control Investment Notifications via /rss Monitor system status Resource pooling, load balancing View partial results of builds in progress Automated build and test for one build tool, one machine Run multiple builds simultaneously Cancel jobs On-demand job invocation via Web Scheduled jobs Trigger builds after check-ins Database of build results Web-based access to results Features Multiple jobs run Multiple servers, remote invocation on one resource simultaneously Run job steps in parallel Single job can use multiple resources Resource selection criteria

43 Build Your Own? Better reporting Investment Notifications via /rss Monitor system status Resource pooling, load balancing View partial results of builds in progress Scheduled jobs Trigger builds after check-ins Automated build and test for one build tool, one machine Database of build results Web-based access to results Run multiple builds simultaneously Features Multiple jobs run Multiple servers, remote invocation on one resource simultaneously Run job steps in parallel Single job can use multiple resources Cancel jobs On-demand job invocation via Web Resource selection criteria Tools for extracting data from logs Annotate builds after completion Customizable reports Trend reports Cross-product reports Resource utilization reports

44 Build Your Own? Better error handling Investment Scheduled jobs Trigger builds after check-ins Notifications via /rss Automated build and test for one build tool, one machine Database of build results Web-based access to results Run multiple builds simultaneously Monitor system status Resource pooling, load balancing Features Multiple jobs run Multiple servers, remote invocation on one resource simultaneously View partial results of builds in progress Run job steps in parallel Single job can use multiple resources Cancel jobs On-demand job invocation via Web Resource selection criteria Tools for extracting data from logs Annotate builds after completion Customizable reports Trend reports Cross-product reports Detect resource failures Resource utilization reports Retry steps after errors/ failures Timeouts for runaway job steps Logging and error reporting

45 Investment Build Your Own? Scheduled jobs Trigger builds after check-ins Automated build and test for one build tool, one machine Developer access to production builds Shared access by multiple teams LDAP authentication Notifications via /rss Database of build results Role-based access control Web-based access to results Run multiple builds simultaneously Resource pooling, load balancing Features Multiple jobs run Multiple servers, remote invocation User impersonation, password management Priorities Monitor system status on one resource simultaneously View partial results of builds in progress Run job steps in parallel Single job can use multiple resources Handle multiple independent groups Cancel jobs On-demand job invocation via Web Resource selection criteria Tools for extracting data from logs Annotate builds after completion Customizable reports Trend reports Cross-product reports Detect resource failures Resource utilization reports Retry steps after errors/ failures Timeouts for runaway job steps Logging and error reporting

46 Investment Build Your Own? Scheduled jobs Trigger builds after check-ins Automated build and test for one build tool, one machine Developer access to production builds Shared access by multiple teams LDAP authentication Notifications via /rss Database of build results Role-based access control Web-based access to results Run multiple builds simultaneously Resource pooling, load balancing Features Multiple jobs run Multiple servers, remote invocation User impersonation, password management Priorities Monitor system status on one resource simultaneously View partial results of builds in progress Run job steps in parallel Single job can use multiple resources More programming features Cancel jobs On-demand job invocation via Web Resource selection criteria Tools for extracting data from logs Annotate builds after completion Extensibility Customizable reports Trend reports Cross-product reports Parameters Conditional steps Detect resource failures Resource utilization reports Retry steps after errors/ failures Timeouts for runaway job steps Modular, composable process steps Control environment for jobs Logging and error reporting

47 Investment Build Your Own? It Never Ends Scheduled jobs Trigger builds after check-ins Automated build and test for one build tool, one machine Developer access to production builds Shared access by multiple teams LDAP authentication Notifications via /rss Database of build results Role-based access control Web-based access to results Run multiple builds simultaneously Resource pooling, load balancing Features Multiple jobs run Multiple servers, remote invocation Support multiple languages and build tools User impersonation, password management Priorities Monitor system status on one resource simultaneously View partial results of builds in progress Run job steps in parallel Single job can use multiple resources SCM integration, bill of materials Portable across hardware/os platforms Cancel jobs On-demand job invocation via Web Resource selection criteria Tools for extracting data from logs Annotate builds after completion Web interface for editing processes Customizable reports Trend reports Cross-product reports Command-line interface Extensibility Parameters Conditional steps Detect resource failures Resource utilization reports Retry steps after errors/ failures Timeouts for runaway job steps Modular, composable process steps Control environment for jobs Logging and error reporting

48

49 Q&A Please ask, or me For more information: Visit our website: