Key Attributes and Responsibilities of a Test Manager Chris Comey SIGIST December 2015
The Test Managers World Standard Test Management Activities Contextual Test Management Activities Test strategy & planning Stages & phases of testing Functional & non-functional Progress monitoring & reporting Test team size, skills, location? Development lifecycle Communication Plan Regression testing Defect management Key Project Processes Industry sector People management Tools in use Resourcing & Budgets This is not an exhaustive list! Projects or maintenance? Compliance, legal & regulatory Product types & source Integrating the test team Issue resolution practices Triggers for change
Key Role of the Test Manager Create awareness Realistic Expectations In You Build Confidence Stakeholder Management In Your Team Testing needs/ dependencies Control ALL Testing Aspects Visible Success In the test function Manage changing dependencies Manage Conflicts Manage the Test Team Provide Solutions Offer/Discuss Options Stand Firm when needed Own your Testing
Tips and areas to concentrate some effort Be ready for the following statements/questions. So long as you are testing everything! How have you prioritised? Where are we with the testing? When will we be finished? Why was this bug missed during testing? Why does testing cost so much? A bit of work up front will prepare you to respond appropriately. Stakeholder confidence in you and your team is not a given!
Stakeholder Map (typical example) Identify which stakeholders need to know what information, and plan your communications accordingly! IT Director Programme Manager Business Managers IT Support Managers Project Manager Programme Test Manager Release Manager Change Manager Test Manager Dev Manager Functional Test Team Non-Functional Test Team Test Env Manager Defect Manager
Deciding what to test first Are you testing projects or maintenance releases? Risk should be the driver for scope, based on what you know! Be ready to explain your reasoning in a few simple principles or statements, e.g. Based on the requirements we have prioritised as follows. We held a risk workshop and this is what was agreed.. Based on the previous defect metrics we have ordered our execution like this. Any other logical approach at release level such as Change Impact Analysis..
Change Impact Analysis Any development release may include Change Requests (CRs) or Defect fixes resulting in.. Testing New/changed functions or functionality New/changed interfaces between functions or systems New/changed data exchange between functions or systems Regression Testing Unchanged functions, functionality or data Unchanged system interfaces or system data exchanges The overall system / network functionality (end to end) All of the above may need to be tested!
Agreeing Test Scoping How Test Coverage looks (typically) Agreed test scope Critical test scope Potential Test Scope We can never test everything, we know there are areas we must test and everything else is up for negotiation! Where we draw the line for the agreed scope is where Risk Based Testing comes in. A workshop with technical and business experts can help define the scope within the constraints that apply (Time, cost, etc.)
Agreeing Test Scoping = Defect So, if we execute the agreed test scope we should find the red & blue defects but not find the black defects Agreed test scope Critical test scope Potential Test Scope Risk based testing does not claim to find ALL defects but should find more high severity defects as the test effort is focussed on the technically difficult areas and business critical areas
But what about the ones we missed Agreed test scope Critical test scope In hind sight we may analyse the facts and say we should have found defect, or that was important enough to increase the test scope Potential Test Scope
Look How Test Coverage looks (typically) Agreed test scope Critical test scope Potential Test Scope To adjust the coverage as shown is easy once you know where the issue lives, so the next time you do the same thing you check for the issue Unfortunately we do not do the same thing very often and therefore cannot target in advance unless
Agreeing Test Scoping How Test Coverage looks (typically) In reality to catch we would need to increase the test scope as shown (possibly) Agreed test scope Critical test scope In real terms it may be more cost effective to keep the testing to the agreed scope and deal with the issue in production (if it actually exists). Potential Test Scope Analysis of production defects, where they are sourced and what would have been required to find them in test, is a valuable exercise. We can look for patterns and include those areas as higher risk.
Execution Scheduling The order of test execution is primarily driven by risk (highest risk tests first). Consider test effectiveness, efficiency and timing. Risk & Likelihood factors may indicate that the tests are executed as follows: Retest defect fixes first, then Execute tests for new code/functionality, then Regression test unchanged code Or maybe some other order! Ask Why does this release exist? Do you have defect metrics? Are your regression packs automated? Should I stop execution if I find a critical bug?
Integrating Testing to the delivery life cycle Iterative releases require cumulative regression testing! Key points: The project Development Lifecycle for this project follows a loose Agile methodology with sprint deliveries providing new functionality and updates to previously delivered functionality within each release. Defect fixes will also be delivered on top of the already delivered functionality.
Progress Reporting We can report what we know What we should know What is complete? What remains to be done? How many defects are currently open and what severity? Are we on track against our plan/schedule? What is our defect find rate against our defect fix rate? What is our defect turn around timeframe in Dev (typically)? Can we predict what to expect? Will we land within budget? What is your confidence level we will complete on time? When do we expect to have executed all tests at least once? How many more releases will we need? How many more defects are we predicting?
Resource Management
Reporting Progress using graphs/tables
Reporting how much more development work is remaining DATA Current Critical/Major Defect Count 100 Current Medium/Minor Defect Count 100 Predicted Critical/Major Defect Count (yet to be found - estimated) 50 Predicted Medium/Minor Defect Count (yet to be found - estimated) 50 Average estimated time to fix a Critical/Major Defect (hrs) 2 Average estimated time to fix a Medium/Minor (hrs) 1 Number of developers working on defect fixing 2 Days per week that developers are working 5 Calcs Hours Days Current Critical/Major Defect Count effort required 200 33.33 Current Medium/Minor Defect Count effort required 100 16.67 Predicted Critical/Major Defect Count effort required 100 16.67 Predicted Medium/Minor Defect Count effort required 50 8.333 Total Critical/Major Defect Count effort required (Actual + Predicted) 300 50 Total Medium/Minor Defect Count effort required (Actual + Predicted) 150 25 Total Effort required 450 75 WEEKLY WORK RATE (# developers * 6 hours * # days per week) 60 10 So estimated time elapsed to complete the dev work with current resourcing levels is 7.5 weeks Work with the developers not against them, but the test teams needs must be taken into consideration!
23-Sep 25-Sep 27-Sep 29-Sep 01-Oct 03-Oct 05-Oct 07-Oct 09-Oct 11-Oct 13-Oct 15-Oct 17-Oct 19-Oct 21-Oct 23-Oct 25-Oct 27-Oct 29-Oct 31-Oct 02-Nov 04-Nov 06-Nov 08-Nov 10-Nov 12-Nov 14-Nov 23-Sep 25-Sep 27-Sep 29-Sep 01-Oct 03-Oct 05-Oct 07-Oct 09-Oct 11-Oct 13-Oct 15-Oct 17-Oct 19-Oct 21-Oct 23-Oct 25-Oct 27-Oct 29-Oct 31-Oct 02-Nov 04-Nov 06-Nov 08-Nov 10-Nov 12-Nov 14-Nov Reporting - When will we be finished? 35 30 25 20 15 10 5 0 Critical/Major defects Total Critical/major defects currently Open New Critical/Major this period Critical/Major closed this period Critical/Major ready for retest (re-test/fixed) Medium/minor defects 70 60 50 40 30 20 10 0 Total Medium/minor defects currently Open New Medium/minor this period Medium/minor closed this period Medium/minor ready for retest (re-test/fixed)
Why didn t we find this during testing?
What is your response? Be ready! There are many reasons why defects may arise in production that were not found during testing! This area/combination was out of the agreed test scope We actually have a defect open on that! We have a result showing a pass for that test so maybe: A different version was deployed to production than we tested! An environmental difference is responsible A data difference has caused the issue User/tester working differences have highlighted the issues (process order) The code implementation/config process for test was different to production We didn t regression test that area for the production release Maybe it was that last minute small change you put in! It was in scope and we missed it! We are not perfect, but Defend your Team until investigations are complete then report
Why does testing cost so much and always over run? 1. Because your requirements are rubbish! 2. Because the developers are rubbish! 3. Because the PM had un unachievable plan! 4. Because we always need more releases than planned (RM?) 5. Because we went in with a low test estimate to win the work/make the business case and knew CRs would come along later! 6. Because the PM spent the test contingency on something else! OR 1. Because testing is an iterative process, more releases=more testing 2. Because Application Development is a difficult and time consuming activity, complex solutions need thorough testing. 3. Experience dictates that issues arise with data, non functional attributes and system integration. It is not only about the functionality 4. Because it is difficult to envisage what a solution will look like until the users can actually use it, they often identify a raft of changes late in the process triggering further releases
Becoming a Test Manager - Comfort zones Comfort zone Stress zone Panic zone Based on a presentation by Erik Boelen
Comfort zones If you stay in your comfort zone (done it before, easy, no problem) You will not feel challenged You will not learn, grow or develop You will not contribute as much as you could You will not gain the respect of your colleagues If you push into the Stress zone (new, makes you think, challenged) You will feel challenged You will have to think and work hard to achieve You will learn and grow in confidence You will increase your comfort zone If you push into the Panic zone (out of depth, don t know what to do, you risk failure) You will likely fail as you do not know what to do You do not have the skills or experience to make the correct decisions
To become a Test Manager you need to push your boundaries Comfort zone Stress zone Panic zone But do so in a responsible way!
In Summary if you want to be a test manager, then Try and secure a mentor (an experienced Test Manager would be good!) If you are working within an organisation then express your interest in test management and raise it at your next performance review meeting (Objectives?) ISTQB Advanced Test Manager Training Course Expand your experience in the areas where test managers live and work! Ask if you can run a test team meeting or defect review meeting Read about or talk to practitioners in other disciplines such as Release Management, Change Management, Configuration Management etc. to understand their role and what it brings to an organisation (identify the test dependencies) Use the Internet to research e.g. UKTMF meets quarterly ( 20)
In Summary if you want to be a test manager, then Ask your manager for test co-ordinator responsibilities such as: Performing defect management (or updating the defect process, if required) Test Environment/Data management activities Take responsibility for test planning and control for a smaller project Take on analysis and reporting of test progress and quality status for a project Ask for exposure to budget information Develop a detailed test plan (Gantt) with deliverables, milestones, dependencies and resources be ready for robust challenge! Complete a Test Exit Report Perform defect analysis, look for patterns and report.
A Test Managers greatest asset? Make sure you take time to think! You don t need to have all the answers! You do need to know all the questions! The more experience and access you have to people, resources and information, the easier it becomes.
Key Attributes and Responsibilities of a Test Manager Testing Solutions Group Ltd 14-16, Dowgate Hill London EC4R 2SU office: +44 (0)20 7 469 1500 mobile: +44 (0) 7711 080697 ccomey@testing-solutions.com www.testing-solutions.com