"A PROCESS FOR SCOPING AND PLANNING TEST AUTOMATION PROJECTS"

Transcription

1 BIO PRESENTATION W8 5/16/2007 1:45:00 PM "A PROCESS FOR SCOPING AND PLANNING TEST AUTOMATION PROJECTS" Jennifer Seale Nationwide Insurance International Conference On Software Test Analysis And Review May 14-18, 2007 Orlando, FL USA

2 Jennifer Seale Jennifer Seale holds Master s degrees in English and Computer Science as well as active CSTE, PMP, and Mercury CPC-QTP designations. She has seven years of experience in the software industry as a developer, tester, test coordinator, and test automation solution architect. Jennifer is a top-rated adjunct faculty member at Franklin University and has taught a variety of undergraduate programming classes over the past four years. For the past two years Jennifer has contributed to the development of the Test Automation Center of Excellence at Nationwide Insurance.

3 Test Automation Centers of Excellence Jennifer Seale

4 Impediments to Automation Projects Lack of experience and knowledge Lack of resources Environment instability Project Process Immaturity Costs Initial investment may be large (tools, knowledge, training) Return on Investment may be long term Maintenance required 2

5 Nationwide s Solution Centralized Test Automation Team Experts in designing automation solutions and using automation tools Partner with projects Provide various services 3

6 Developing a Center of Excellence Set up Shop Document Review Process Process for engagements Templates to support process Process to manage requests for services Process to track status Continuous Improvement Best Practices and Standards 4

7 Customer Engagement Process (CEP) Six Phase Process 1. Scoping 2. Planning 3. Development 4. Execution 5. Wrap Up 6. Maintenance 5

8 Templates Supporting our Process Initial Screening Survey Readiness Assessment Complexity Assessment Statement of Work (SOW) Schedule Cost Benefit Analysis (CBA) Training and Knowledge Transfer Plan Proof of Concept (POC) Customer Development Plan Test Automation Design Document Test Automation User Guide Project Log 6

9 CEP Phase 1: Scoping Learn about project and needs Complete Assessments Determine scope of engagement Develop preliminary schedule Conduct POC Get approval to continue to planning 7

10 Scoping: Assessments Initial Screening Survey Does automation make sense for the project? Focus on goals, application, environments, builds Readiness Assessment Is the project ready for a test automation effort? Focus on resources, processes, artifacts, funding Complexity Assessment How much effort will it take to automate this project? Focus on scope, dependencies 8

11 Scoping: Determine Scope We typically use the following method to determine the scope of the project: 1. Determine automation needs. 2. Break down automation needs into a list of welldefined, manageable units of work. 3. Prioritize the list in terms of functional dependencies. 4. If the prioritized list is long, divide the list into logical sections. These sections will be used during planning to allow a phased approach to automation development. 9

12 Scoping: Preliminary SOW Using template as base, add details as they are revealed through conversations with the customer. Helps to have information in writing Will become a complete description of all the work involved in the automation effort. Need to determine preliminary schedule to help in resource planning. 10

13 Scoping: Proof of Concept (POC) A POC is a prototype of the automation that demonstrates the technical feasibility of the automation project. Use Complexity Assessment results to help determine an interesting subset of scenarios to cover in POC. Automate this subset of test scenarios using the selected automation tool(s) and report results. 11

14 Scoping: Benefits of a POC Increased confidence in the project Early identification of challenges and risks Early set up of automation environment Provides useful input to estimating effort 12

15 Scoping: Customer Development Plan When we say No to an automation project, we offer to provide the customer with a document that outlines what needs to be addressed on their project before an automation effort can begin. Most useful when: the project is interested in automation automation makes sense for the project other factors exist that put the success of an automation effort at risk 13

16 CEP Phase 2: Planning Refine SOW Refine Schedule Perform Cost Benefit Analysis Training and Knowledge Transfer Plan Test Automation Design Document Approval for SOW 14

17 Planning: Refining SOW Finalize communication plan Finalize details of objectives and deliverables Finalize details of schedule Finalize cost estimates 15

18 Planning: Cost Benefit Analysis Using template as a base, tailor to fit project specifics. Estimates all costs associated with automation effort. Predicts benefits of automation effort based on how often the project will run the automation once it is complete. Predicts the breakeven point, when the benefit is equal to the cost. 16

19 Planning: Automation Costs When planning for an automation, consider the following costs: Staff Resources Training Learning curve Initial Development Execution and Analysis Maintenance Enhancements Hardware Support 17

20 Planning: Automation Benefits There are three types of benefit that can be realized through Test Automation: Reduction benefit realized when a test suite that is currently being executed manually is replaced by an automated suite. Coverage benefit realized when an automated suite is created or extended to perform testing that is not currently being executed manually due to lack to resources or funds. Quality benefit realized by detecting defects earlier in the software development life cycle due to the automation. 18

21 Return on Investment Generic Automation Savings Profile Initial Investment Coverage Reduction Quality Jan-06 Mar-06 May-06 Jul-06 Sep-06 Nov-06 Jan-07 Mar-07 May-07 Jul-07 Sep-07 Nov-07 Jan-08 Mar-08 May-08 Jul-08 Sep-08 Nov-08 Jan-09 Mar-09 May-09 Jul-09 Sep-09 Nov-09 Break Even Hours Months

22 Planning: Training and Knowledge Transfer Plan It is important that we prepare key members of the project team to take over the automation. We tailor this plan specifically to meet the needs of the project. Often this plan contains its own schedule. Typically produce a Test Automation User Guide. 20

23 Planning: Test Automation Design Document Documents the automation design for the project. Provides an overview of exactly what was automated for Test leads to use when planning future testing efforts. Documents all design decisions and workarounds that were employed. 21

24 Approval of SOW The completed SOW is sent to the customer stakeholders for review. We conduct a face-to-face meeting where we walk the customer through the SOW to ensure we are in agreement. Once the SOW is approved, development begins. 22

25 CEP Phase 5: Wrap Up Complete Training and Knowledge Transfer and Plan Update SOW with results of engagement SOW Complete approval meeting Customer Satisfaction Survey Engagement Retrospective 23

26 CEP Phase 6: Maintenance Follow up with projects every 6 months for life of project Update CBA with actuals Address noteworthy issues 24

27 Managing Requests for Services Spreadsheet of requests established Tab per request with overview page Log all pre-engagement contact Used for tracking and forecasting Available on shared drive, all team members responsible for keeping upto-date. 25

28 Tracking Status: The Project Log Spreadsheet with multiple tabs Tab per document to track change requests Tab to track technical issues Tab to track resource issues Tab for lessons learned Always available for review Wealth of historic data for future projects 26

29 Continuous Improvement Document Change Request (CR) System Survey Results Lessons Learned Retrospectives Quarterly reviews to address CRs and improve the process and associated templates 27

30 Best Practices and Standards Perfectly positioned to tailor theoretical best practices to specific business Reuse solution models across applications Develop standards for specific technologies used by the business 28