Test Strategy Evolution Throughout the Lifecycle. Chris Comey & Davidson Devadoss

Transcription

1 Test Strategy Evolution Throughout the Lifecycle Chris Comey & Davidson Devadoss

2 We no longer ridicule Darwin Are Evolution principles relevant and applicable in the modern IT work space? Survival of the fittest What is the fittest? In hindsight, that which survives Is this relevant for software testing?

3 Why Test Strategies Exit 1) To define a test framework & approach 2) To ensure the key activities are considered 3) To integrate the test function with the other teams 4) To ensure testing needs are highlighted 5) To communicate and raise awareness 6) To set ownership, roles & responsibilities for critical tasks 7) To gain agreement for the above sign off 8) To address RISK!

4 Who works in a static environment? The world changes around the project Change is driven by technology, integration, changing requirements and test results Assumptions made are proven or disproven Motivational factors and business priorities change

5 What are testers good at? Analysis Attention to detail Ability to adapt to changing priorities Focusing on priority Assessing risk and scoping test coverage Reporting time, cost, quality

6 Example 1 : Integrating Legacy Systems Test Strategy developed I/F New I/F I/F Legacy

7 Delivery Strategy Development Team (x2) Support Team Infrastructure Design Next version under build Test Team Latest version under system test Test Team Q-Gate Build env Deploy code Shakedown Q-Gate Sys Test Complete C/M defects fixed Good enough 2 Integrate Integration Support env Integrated Environment Integration Testing

8 Code Delivery Strategy Rel 3 Next version under build Rel 2 Rel 1 Test Team Environments Integration Support Env Integration Testing

9 What went wrong? Initial issues stabilising the first environment, meant: System Testing did not start as scheduled Development started to put releases on the shelf test backlog Pressure to deliver to the Integration stage started to build First release not fit for testing to start, 2 nd dev release ready, 2 nd environment being prepared Analysis showed: Decision 3 teams delivering individually (infrastructure, application code, Content, integrating to stable legacy interfaces ) No Link testing Focus on getting 1 release ready for Integration (warts and all)

10 Some progress! One release tested enough to go for integration Still struggling to build 2 nd (and now 3 rd ) system test environment Observations No link testing meant infra, new code and new data content were not integrating prior to system test starting (so no test progress) Ownership of defects being disputed (code, env, config, etc) No/poor release notes Release content mismatches found in testing No configuration management meant stable environments could not be reproduced from scratch Double/treble keying of fixes required Test Environment support provided by Live support (when available!) version 1/2/3 under build version 1&2 under system test V1 Integration Support env V1 Integrated Environment

11 Observations 1) Problems were seen by management as being issues within testing, when they were actually much wider! 2) The test strategy looked inwards at testing within the test team, it did not define the dependencies on other teams in enough detail 3) Testing relies heavily on the basic processes being in place release management, environment management, change management, configuration management, defect management, all of which need to be agreed with the relevant parties. 4) Due to the number of teams involved there was a reluctance to deviate from the agreed test strategy we would have to go through sign off again! 5) There appeared to be a reluctance to do the right thing, only what had been agreed (if I do what was agreed I am safe) 6) A good test strategy may not be the right test strategy 7) There was a lessons learned document from a previous project that detailed nearly all of the issues being experienced!

12 Example 2 : Phased release offshore Development Project Objectives: To deliver a new system to an extremely demanding client that is not prepared (too busy) to fully document their requirements. Key points: Off shore development team Project approach to provide incremental deliveries for a series of solution show & tells to ensure the solution is fit for purpose (Agile delivery).

13 Project Test Stages Unit Test Developer Component Testing Independent Developer Component Integration Testing Lead Developer Smoke Test (QA Test env) Lead Developer Functional System Testing Test Team Basic Functionality System Integration Testing Test Team Critical End to End Scenarios Non Critical End to End Scenarios Non Functional Testing Test Team UAT Stage Business Team

14 The identified Risks at the Mid point review

15 Identified risks pre System Test stage Component Integration Testing Risk 1. There is a risk here that what works in Dev QA may not work in QA Test this would extend the Smoke testing stage in QA.

16 Identified risks & mitigating actions Smoke Test Risk 2 There is a risk that the source of any issues will be difficult to identify as Development do not have access to QA Test environment due to data sensitivity issues. This could mean that items that work in Dev QA not in QA Test are difficult to isolate. but

17 Identified risks & mitigating actions Development Testing overall Risk 3 There is a risk that the Dev testing will complete successfully but critical issues will be found when the system is tested in an E2E manner. This is due to the lack of documented Business Processes which detracts from system context for development of the system.

18 Identified risks & mitigating actions Risk 4 There is a risk that Dev will use all the time allocated for testing in order to deliver a higher quality product when the project would have preferred to take the delivery earlier with a lower intrinsic quality level. Potentially this means higher priority E2E defects will be raised later in the lifecycle potentially endangering the project end date.

19 Identified risks & mitigating actions Risk 5.There is a risk that the project will be delayed striving for perfection and delivering low priority fixes in numerous releases that detract effort from the important defects and risk introducing new higher priority defects. This will prevent the stage from completing and the next stage from starting.

20 Identified risks & mitigating actions Critical End to End Scenarios Assuming Risk 3 has been mitigated earlier in the lifecycle then any issues here will be as a result of environmental issues, deployment or configuration issues or True E2E integration issues. Non Critical End to End Scenarios By definition any issues discovered during test execution of lower risk scenarios are lower impact. As all defects will be assessed and categorised these will be dealt with in the normal manner.

21 Identified risks & mitigating actions UAT stages Risk 6 There is a risk that the business users will not accept the way in which the system fulfils the business requirements. This would trigger Change Requests late in the project lifecycle and is a major risk to delivery.

22 Identified risks & mitigating actions Risk 7 There is a risk that the user will identify a new or missing business processes during UAT that will trigger a change request late in the life cycle.

23 What was learned? 1. Test Strategies should be written for the situation 2. A good strategy may not be the right strategy! 3. The world changes around the project, your strategy may have been right but is now incorrect due to changing external factors 4. Good practice put in place at the start of the project can have a negative impact later if not revisited and assessed in the current situation 5. Even if you involve the users at each stage this does not guarantee success

24 What was learned? 6. Detail the testing dependencies and inputs required (specially if they are outside your control) 7. Lessons learned are not learned if the learning is not applied! 8. Ensure the strategy complements the project objectives rather than replacing them 9. Schedule Test Strategy review points in the project plans the world changes and so should we 10. Use the RBT approach to validate the test strategy throughout the test lifecycle

25 Questions? Who has worked on a programme or Project that has been scrapped? Was that a project failing or changing external factors? Don t be a Dodo!

26 TSG Bringing Value to: Organisations and People; Systems and Developments; Business & Corporate Objectives Testing Solutions Group Ltd 5 th Floor Houndsditch London EC3A 7BT Tel: +44 (0) Fax: +44 (0) Web: