"Session-based Exploratory Testing on Agile Projects"

Transcription

1 T4 Class 4/19/2012 9:45:00 AM "Session-based Exploratory Testing on Agile Projects" Presented by: Bob Galen Deutsche Bank Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL

2 Robert Bob Galen Deutsche Bank Global Technology Bob Galen is an agile coach at Deutsche Bank Global Technology and director, agile solutions at Zenergy Technologies, a North Carolina-based firm specializing in agile testing and leading agile adoption initiatives. Bob regularly speaks at international conferences and professional groups on topics related to software development, project management, software testing, and team leadership. He is a Certified Scrum Master Practicing (CSP), Certified Scrum Product Owner (CSPO), and an active member of the Agile Alliance and Scrum Alliance. In 2009, Bob published Scrum Product Ownership Balancing Value from the Inside Out, which addresses the gap in guidance toward effective agile product management. You can reach Bob at bob@rgalen.com or bob.galen@zenergytechnologies.com

3 Session Based Exploratory Testing on Agile Projects Lessons Learned at icontact from Bob Galen VP & Agile Coach, Deutsche Bank Global Technology President & Principal Consultant RGCG, LLC Introduction Bob Galen Somewhere north of 20 years experience Various lifecycles Waterfall variants, RUP, Agile, Chaos, etc. Various domains SaaS, Medical, Financial, Computer Systems, and Telecommunications Developer first, then Project Management / Leadership, then Testing Pieces of Scrum in late 90 s; before Agile was Agile Lucent in using Extreme Programming Formally using Scrum since 2000 Currently Agile Coach at Deutsche Bank Global Technologies in Cary, NC Connect w/ me via LinkedIn if you wish Bias Disclaimer: Agile is THE BEST Methodology for Software Development However, NOT a Silver Bullet! Copyright 2012 RGCG, LLC 2 1

4 Outline Introduction Exploratory Testing and SBET icontact Background History of our use of SBET Some key pro/con of our evolution Release mapping Q&A Copyright 2012 RGCG, LLC 3 Exploratory Testing Basics What is Exploratory Testing Basic dynamics? Principles? Practices? How do you prepare? How do you document the testing? Pairing? Agile implications? Copyright 2012 RGCG, LLC 4 2

5 Exploratory Testing Session Based What are the dynamics surrounding sessions What testing do you focus the sessions on? Time-boxes? Focus guidance Charters? Guidelines? Documentation? De-brief, discovery sharing? Agile Implications? Copyright 2012 RGCG, LLC 5 Why is Session Based Exploratory Testing (SBET) so applicable to agile teams? We ll explore the travels we had at icontact in implementing and executing Exploratory Testing, then wrapping it with Sessions Nimble handling the most critical aspects of the day; changing scope & requirements are relatively easily handled Reactive always testing the most important thing Collaborative focus on people and skill rather than completely defined test cases; lightweight; pairing; fosters a whole-team view Resilient to change, to operating conditions (in development environment, interfaces, databases), to what s working and what s not Simple & Fast to ramp-up, to begin testing, to train new hires, to document Copyright 2012 RGCG, LLC 6 3

6 icontact Background Company formed in late 2003, so approximately 9 years old SMB Marketing Singular SaaS product; small vs. medium business delineation; w/options LAMP technology stack; heavy PHP use in middle and lower tiers Added testers to the teams in early 2009 So from , development-driven testing Scrum shop from late 2007 Cross-functional agile teams introduced early in 2009 Mid mature Scrum shop making monthly deployments of customer value Early 2012 Kanban shop focusing on Continuous Deployment Copyright 2012 RGCG, LLC 7 Initial steps Early 2009 early 2010 We started with a small, singular script smoke test that was being run on a periodic basis Adding a few new, functional tests every sprint Both/all were manually executed Then we broke the smoke test up into Charters and began to run SBET as a means of Distributing the testing beyond a lone tester, across teams, and to gain transparency Isolating Charters towards specific functionally focused Scrum teams (Create, Integrate, Track, Core, Contacts) Reduce execution time and foster test pairing Copyright 2012 RGCG, LLC 8 4

7 Follow-on steps Early 2010 Late 2010 Charters and SBET were our primary testing vehicle; that and collaboratively driven functional testing within our scrum teams The # of Charters began to grow Aligning towards each product team As we added / extended functionality Continued to expand our coverage Physically grow as a company We had always been a paired testing environment Copyright 2012 RGCG, LLC 9 Paired SBET From the beginning we felt that Exploratory Testing should be a paired activity Developer + Developer Tester + Tester Developer + Tester Team member + Customer Support member Cross-team implications As we continued to leverage continuous SBET, the paths got somewhat old ; we needed new eyes & new pairs Teams tested each others work; ran each others Charters Side-effect that teams needed to x-communicate more; heavily Wiki based Copyright 2012 RGCG, LLC 10 5

8 Paired SBET Benefits The impact of inviting customer support (and other x-functional team members) was HUGE In understanding customer usage In adding / changing Charters and focusing domain knowledge Developers increased their understanding & empathy for testing It s Hard! It s Long! We need to leverage our unit test investments It needed broader automation! Copyright 2012 RGCG, LLC 11 Early 2011 Evolution We re de-emphasizing SBET as our do all testing method. Focus it on: Smoke Tests, for example after a hot-fix Sprint-based Test Fests that are whole-team based Risk-based regression testing Starting to write our test cases in Cucumber step format; so that we can easily convert them / run them as Cucumber tests So in each sprint we develop functional tests AND adjust Charters for SBET as appropriate We do a Test Plan per team that shares / exposes the Sprint x Sprint and Release level testing strategy Building a repository of these functional tests Copyright 2012 RGCG, LLC 12 6

9 Late Latest Evolution Converting our Charters to functional automation We ve used the expanded Charters as a base for developing mostly Cucumber based automation Found the expanded format helpful for the conversion Reducing the # of steps and information in the Charters Converting them back to a truer form of SBET Charter with little leading information Couple with our Test Idea brainstorming / design sessions to drive more new tests Re-analyzing some of our results; leading towards updates of our testing approaches & artifacts Copyright 2012 RGCG, LLC 13 General Benefits of SBET It was a quick way to begin testing in an agile context From scratch; w/o any formal test repository / base Inherently collaborative Inherently agile, context-based, risk-based; testing the most important things of the moment Pairing worked wonders to foster a Everyone Owns Quality/Testing mindset Empathy for testing challenges Instilled the right behaviors Pulled the client / customer facing folks into our teams Copyright 2012 RGCG, LLC 14 7

10 Regression Dynamics of SBET Two-edged sword when using it for Regression On the one hand, the Charters get beefier and more prescriptive Logging looks very similar code drop over code drop Can manage, report, and control x-team execution But It s really not the core intent of ET we re controlling the exploration No matter what you do, it s not repeatable Can de-rail your Endgame from a defect discovery perspective In hindsight, probably not the best use Copyright 2012 RGCG, LLC 15 Planning & Reporting Each team is responsible for planning their Charters for a given set of sessions. Planning occurs within Sprint Planning Testers define a Test Plan per sprint point is to drive strategic thinking within the team Identify and x-team dependencies for testing Test Fests per sprint & per release Clearly defined strategies (why) behind the testing focus is team-driven Results simply visible on the wiki as charters / sessions are completed War room to coordinate items found, questions, sharing, issues, etc. Copyright 2012 RGCG, LLC 16 8

11 The Agile Release Train Synchronized Internal Release External Release Team 1 Harden Team 2 Team 3 Continuous Harden Continuous Harden Integration Integration X-team Harden Docs, Training, Support, UAT, Comp. Continuous Integration Team 4 Team n Harden Continuous Integration Copyright 2012 RGCG, LLC 17 The Agile Release Train Example: icontact / SaaS Model 3 weeks / 15 days 4-5 days SBET, Exploratory Regression Testing Production Release Team 4 Team 1 Team 2 Team 3 Team 10 Environment Evolution Continuous Integration Continuous Integration Continuous Integration Continuous Integration Harden Harden Harden Harden X-team Harden Docs, Training Rinse & Repeat Dev + QA QA -> Staging Production Copyright 2012 RGCG, LLC 18 9

12 Retrospectives Leverage the teams Retrospectives as a place to discuss: SBET testing strategy Timing Coverage X-team interactions At a scrum team and release level. Reflect on our results and implications to Functional tests (additions, retiring, new suites, etc.) Extending our Charters Copyright 2012 RGCG, LLC 19 Test Ideas We ve recently begun to use Rob Sabourin s views towards Test Idea generation - Brainstorming Formation Prioritization Execution & Retrospective Very useful to dovetail the ideas to the execution within Charters Sort of an idea laboratory for your ideas Fast feedback for fine-tuning Copyright 2012 RGCG, LLC 20 10

13 Wiki-based examples of our SBET artifacts We leverage a wiki (Confluence) for our SBET charter storage, session notes, defects, and repetitive run logs We build in some forward learning notes, where applicable, for our retrospectives So here are some examples Copyright 2012 RGCG, LLC 21 Overall Charter - Findings Each charter has a wiki page associated with tracking results. Results would be captured per session including very specific bugs, notes, and concerns / questions. Copyright 2012 RGCG, LLC 22 11

14 Charter Team Assignments Team and individual assignments would be made on a TestFest or event basis. This page identified the notion of: Charter Owners and identified groups, individuals, pairs, that were recommended to execute the charters. This varied as our application risk-based test planning drove us in different directions. So very dynamic; but, with some solid core of team familiarity per charter Copyright 2012 RGCG, LLC 23 Charter Test Index / Management This wiki page rolls up all charters for overall test execution. We also used it to monitor execution and connect to individual results. You can see notes regarding Automation Assisted. As we developed functional and other automation, we connected it to the charters & sessions. So automation execution became part of the sessions. Not so to de-focus the exploration, but as a means of augmenting the focus and roundtrip evolution of automation on Charter-based boundaries. Copyright 2012 RGCG, LLC 24 12

15 Individual Charter - Logging Each charter has a wiki page associated with tracking results. Results would be captured per session including: Software version, release target Who executed the charter Bugs found Script of travels in executing this session and specific guidance around (1) how to reproduce bugs and (2) what would be good to convert into functional test cases. Copyright 2012 RGCG, LLC 25 Individual Charter - Logging Each charter has a wiki page associated with tracking results. Results would be captured per session including: Another important aspect of our logging was that all history of execution was in a single wiki page. Yes, this could get unwieldy over time, but we didn t find that to be the case yet What we found is that sessions could be directed based on historical notes on Stability vs. flakiness Previous coverage Previous discoveries Copyright 2012 RGCG, LLC 26 13

16 Individual Charter Expanded Steps Each charter has a wiki page associated with tracking results. Results would be captured per session including: Testing guidance including entry, exit conditions and test steps We expanded on the steps NOT to inhibit exploration, But as an effort to allow the charters and execution notes to allow us to develop more formal functional test cases That would be extracted from the SBET efforts Copyright 2012 RGCG, LLC 27 The Future Next Steps We re moving back to the traditional exploratory nature of ET and Session-Based ET That and team-based Test Idea brainstorming and Agile test planning Are driving our testing creativity and value-added tests Kanban evolution continuous flow Continuous every check-in Off of Trunk and notion of Feature Branches (MMF) Continuous Deployment weekly deployments to production Copyright 2012 RGCG, LLC 28 14

17 The Future Next Steps We re realizing our test automation strategy and gaining significant automation coverage Allowing to next focus on agile performance testing, UX integration, and continuously improving tests Copyright 2012 RGCG, LLC 29 Wrapping up SBET can be a very effective agile approach Beginning / Building team new to agile Ongoing team maturity Be careful not to overload your Charters; allow/support for true exploration Pairing can be powerful Create wiki-based, very simple management & tracking Leverage Charters for your automation development Prioritization & experimentation Driving time savings & risk mitigation Copyright 2012 RGCG, LLC 30 15

18 Bob Galen Principal Consultant, RGalen Consulting Group, L.L.C. Director of Agile Solutions, Zenergy Technologies, Contact Info Blogs: Project Times - Business Analyst BA Times - My Podcast on all things agile - Director, Agile Practices, icontact Scrum Product Ownership Balancing Value From the Inside Out published by RGCG in Experience-driven agile focused training, coaching & consulting Cell: (919) bob@rgalen.com bob.galen@zenergytechnologies.com Copyright 2012 RGCG, LLC 31 16