Risk-Based Testing for Agile Projects PRISMA Erik van Veenendaal www.erikvanveendaal.nl 1
Erik van Veenendaal www. erikvanveenendaal.nl In testing since 1989 working for many different clients and in many different roles Author TMap, ISTQB Foundation and TMMi and many other books and papers Former Vice-President International Software Testing Qualifications Board (ISTQB) TMMi Foundation board member Regular keynote and tutorial speaker Winner Testing Excellence Awards Living in Bonaire (Caribbean island) Table tennis player and coach Improve IT Services BV 2
Testing Challenges Increasing importance and size of software in society as a whole e.g., amount of software in consumer product doubles every 24 months, number of defects is almost constant High Competition & Outsourcing Time-To-Market, Product Quality, Price Levels are essential for business success Testing often takes 30-40% of project costs Risk-based testing deals with both effectiveness and efficiency Improve IT Services BV 3
Agile Silver Bullit 10000 1000 Defec cts 100 10 1 Project Size (KLOC) 10 100 1000 despite continuous growth in adoption of agile, many organizations are still facing the the same challenges as in previous years, World Quality Report Improve IT Services BV 4
What is Risk? A factor that could result in a future negative consequence; usually expressed as impact and likelihood (ISTQB Glossary) Testers only have the responsibility to identify the risks and provide information on their status to dare to undertake management attitude and style.. Improve IT Services BV 5
Testing = Risk Management Objective: most feasible coverage effective usage of limited resources optimize test effort Limited resources staffing infrastructure time!.. The right level and type of coverage on the right parts at the right time Always applicable, whether it is V-model, Agile or... Improve IT Services BV 6
Our Challenge. if only we knew!! Improve IT Services BV 7
Risk-Based Testing Risk identification looks at ways of establishing what the risks are and where they are Risk analysis looks into the critical, complex and potential error prone areas Then we build tests to mitigate the risk Subsequently we monitor and report regarding the risks Improve IT Services BV 8
Based on Practical Experiences 9 Improve IT Services BV 9
Risk Identification Split up test basis in functional or technical items V-model: based on requirements Agile: backlog items / user stories for next sprint Can also be based on a brainstorm session Maximum number of appr. 35 risk items Risk item 1 Risk item 2 Risk item 3 Risk item 4 Functionality Security Functionality Interoperability Improve IT Services BV 10
Risk Analyses Risk = impact x likelihood What is the impact for the business? What is the likelihood that there are defects? Determine factors based on previous projects, e.g., defect patterns Likelihood technical risk Impact business risk You already know this! Exercise: Risk Factors Improve IT Services BV 11
Factors From Practice Likelihood complexity new development (level of re-uses) interfacing size technology geographical spread inexperience (of development team) Impact defect patterns / history business importance ( selling item ) financial (or other) damage (e.g. safety) usage intensity external visibility cost of rework Customization required Improve IT Services BV 12
Risk Assessment in a Nutshell 1) Stakeholder analysis: Assigning Factors 2) Individual scoring they shall make choices Item 1 Item 2 Item 3 Item 4 Business importance 5 54 43 52 Usage intensity Safety 3) Consensus meeting 4) Position the risk items in a Product Risk Matrix 5) Define a differentiated risk-based test approach Improve IT Services BV 13
Product Risk Matrix MoSCoW priorities Likelihood H M Could Test focus of unit testing x x I III x Must Test x II focus of acceptance x testing IV L L Won t Test M Should Test H Impact Improve IT Services BV 14
Play the Card Game: Risk Poker Poker Planning / Wide Band Delphi based Improve IT Services BV 15
Wide Band Delphi-Poker Planning (1) Bottom-up estimation techniques - wide variance in estimates - commitment of people involved - accurate estimation we use the collective wisdom of the team Each estimator has cards with valid estimates A user story (product backlog item) is discussed Each estimator selects an estimate Cards are turned over Discuss differences (especially outliers) Repeat until consensus Improve IT Services BV 16
Wide Band Delphi-Poker Planning (2) Moderator User Story Explain Estimator Estimates Collect OK? Final estimate Share info OK? Improve IT Services BV 17
Risk Poker the same, but Scoring separately for likelihood and impact Discussion shall be factor based Limited number of cards for each value (color) Agreement, Product owner (impact) and leaddeveloper (likelihood) act as tie breaker Improve IT Services BV 18
Let s Play Example User Story As as bank account holder, I want a yearly financial overview of my accounts, so that I can submit my taxes Impact? Likelihood? Impact: Likelihood: Improve IT Services BV 19
Agile Product Risk Matrix example User Story Improve IT Services BV 20 See any issues with this?
Differentiated Test Approach!! Test design techniques Reviews & inspection Test design start-up meetings Reviews of test design Level of detail of test cases Exit (DoD) criteria, e.g., coverage Level of independence More time & effort Most experienced person Priority setting Regression testing Re-testing without this a risk assessment doesn t make much sense!! Improve IT Services BV 21
Real-Life Agile Example Likelihood H Exploratory Testing Support module tests Review module tests M L L Could Test x x I III Exploratory Testing Won t Test Impact M x x Test design techniques Review Must Test design Support module tests Review module tests x II IV Test design techniques Review Should design Test H Improve IT Services BV 22
Agile One Page Test Plan Shall include a definition of done Must Test.. Test Approach.. Should test Test Approach.. Could Test.. Test Approach.. Would Test.. Test Approach A.. 1-2 hour session to achieve a one page Sprint Test Plan Improve IT Services BV 23
Communication Improve IT Services BV 24
Product Risk Matrix Initial Likelihood - Tech hnical Risk US 58 I US II 27 US US 31 III 49 IV Impact - Business risk Improve IT Services BV 25
Product Risk Matrix Progress Likelihood - Tech hnical Risk US 58 I US II 27 US US 31 III 49 IV Impact - Business risk Improve IT Services BV 26
Key learning points A structured and practical approach for risk-based testing is available Make it Agile,, but use the formal rules Efficient, Effective and Attractive Define a risk-based differentiated test approach Practice risk-based tracking & reporting Improve IT Services BV 27
Downloads Go to www.erikvanveenendaal.nl for: A full PRISMA white paper (e-book) Paper: 10 Recommendations on Risk-Based Testing Paper: Product Risk Assessments in Agile Projects (Risk-Poker) To acquire the PRISMA freeware tool sent email Book Practical Risk-Based Testing; The PRISMA Approach, ISBN9-78490-986070 (www.utn.nl) Improve IT Services BV 28
www.erikvanveenendaal.nl erik@erikvanveenendaal.nl @ErikvVeenendaal Improve IT Services BV 29
Any questions...? Thank you!! Improve IT Services B.V. 30