Agile Testing: Challenges still to be conquered. Mobile:

Transcription

1 Agile Testing: Challenges still to be conquered Mobile:

2 Pablo Garcia started as a tester in 1996 for Ericsson. After passing through roles like Test Manager, Project Manager, Program Manager he worked as Total Program Manager managing all the Ericsson Development in India. Now, over 20 years later he has run over 50 assignments besides running his own test companies, he founded the current one in July this year, and aiming to be 10 employees by Christmas. Pablo has spoken at national and International Conferences like NFI, Test management Forum and Star West, and has given Testing courses since Amongst other he has educated over 200 nurses in acceptance testing during the last 5 years.

3 Pablo Garcia started as a tester in 1996 for Ericsson. After passing through roles like Test Manager, Project Manager, Program Manager he worked as Total Program Manager managing all the Ericsson Development in India. Now, over 20 years later he has run over 50 assignments besides running his own test companies, he founded the current one in July this year, and aiming to be 10 employees by Christmas. Pablo has spoken at national and International Conferences like NFI, Test management Forum and Star West, and has given Testing courses since Amongst other he has educated over 200 nurses in acceptance testing during the last 5 years. I like to do things right the first time.

4 Testing: Challenges still to be conquered Personal Accountability. Harder when teams are involved Large Agile Organizations Test in an Agile organisation with 10 teams or more. How to maintain a minimum level of Test and Quality in all teams. Who does what if everyone does everything? Should we have a test Strategy or a Strategist? Test Competence Improvement. Mixed Practices Test in an organisation with Agile teams as well as teams with other practices. Buying Software from a Agile vendor. Pablos process Synchronizing the test effort. Defining Test & Quality Requirements Contract issues Running your acceptance testing,and Agile UAT. The 3 levels of delivery Metrics and Improvements Assessing what has to be improved DDP / DDP+ Other important metrics Extremely important metrics!!!

5 10 or more Agile teams

6 10 or more Agile teams

7 10 or more Agile teams Why so many teams? 1-Large Product 2-Complex Product 3-Lots of interacting systems 4-Other

8 10 or more Agile teams Maturement process 1: Nothing gets released. Time

9 10 or more Agile teams Maturement process 2: Stabilization Sprint Time

10 10 or more Agile teams Maturement process 2: Stabilization Sprint Time

11 10 or more Agile teams Maturement process 3: Test pipeline TP TP TP TP Time

12 10 or more Agile teams Maturement process 4: System test is a support organisation. Providing: -Test environments -Automated tests -Tools -Competence ST Time

13 10 or more Agile teams How to maintain a minimum level of Test and Quality in all teams. Every teams shall have 1 tester. The team without a tester defines the minimum level.

14 10 or more Agile teams How to maintain a minimum level of Test and Quality in all teams. Gathering all the testers. Weekly meetings: Synchronizing Competence Metrics, same for all teams Help eachother Better efficiency It is possible to have ½ a tester per team or less, but very difficoult. A professional tester is a professional tester!

15 10 or more Agile teams Should we have a test Strategy, or a Strategist? A written test strategy is good, but the need of a test strategist is bigger. Runs the weekly meetings Responsible for metrics (over time too) Formulates a test strategy (5-10 years) Competence responsible

16 10 or more Agile teams Test Competence Basic Education Advanced Education Books / White papers Conferences Round table meetings Keep People interested. Competence is a complex word.

17 10 or more Agile teams, Competence Education Experience Mentorship Documentation & Templates Competence Planning

18 Mixed Practices With Mixed Practices Syste m req. System test High level design Integration test Detailed design Unit test Code

19 Mixed Practices Syst em req. High level design Detailed design Code Integration test Unit test System test Syst em req. High level design Detailed design Code Integration test Unit test System test With Mixed Practices, create a Test Pipeline. Synchornize, 2,4,6 weeks. Some customers are not prepared to receive new releases often.

20 Buying Software from a Agile vendor How to buy Systems from an Agile vendor. (from a QA perspective) Pablos Process: 1-How do you test? 2-Create a complete chain of test 3-Define Quality Requirements 4-Put in the contract/contract Issues The 3 levels of delivery Running your Agile UAT

21 1-Ask the vendor the easy question: How do you test? Exploratory Context Driven Structured test TDD finds everything We don t Buying Software from a Agile vendor A book often called xxxxx Test Methodology A well defined process that they follow

22 Buying Software from a Agile vendor 2-Lets synchronize our tests. (vendor and customer) Define a chain of tests. Reviewing Unit tests Functional tests System tests Integration tests UAT 3.Define Quality Requirements (Example).

23 project common Test Strategy Requirements Vendor Customer Customer Users 1.Unit Test 2.System Test Delivery 3.Pre ACC-1 4.Pre ACC-2 5.Acceptance Testing

24 Vendor test activities/delivery and Quality attributes 1.Unit test and Definition Of Done -Code reviewed 100% T.B.D. -Code Statically analyzed 100% T.B.D. -Code Coverage 80% T.B.D. -Source Code Handling process completed 2. System test -Positive Requirement coverage 100% -All prioritized functionality negative coverage -Deployment test done -Business critical E2E Test cases automated with Selenium Delivery includes: -System Test plan -Specified Definition of Done (DoD) for Unit Test -Test cases from System test, and Unit Test if possible -Traceability matrix between Requirements and Test Cases -System Test report -DoD execution checklist -The documentation will follow test documentation standard IEEE829 Quality Attributes: All test activities have been executed according to plan. The system has no Critical or Major bugs. All minor bugs are documented as bug reports. All defined deliverables are included, delivered and up to date on a agreed storage.

25 3.Pre-Acceptance 1 -All deliverables included and delivered on to the right place) -Review all development/test documentation. Customer test activities and Quality attributes Pre-Acceptance 1 & 2. Quality Attributes: All defined deliverables are included in the delivery. 4.Pre-Acceptance 2 -Run the most common E2E user flows, manual at start, automated later in the project. Delivery includes: -Pre-Acceptance 1 and 2 Test plan -Test cases -Traceability matrix between Requirements and User flows. -Pre-Acceptance Test report, including fundamental metrics. -Automated test cases -The documentation will follow test documentation standard IEEE829 Nothing missing or errouneous in the incoming delivery All planned user flows have been executed. The system has no Critical or Major bugs. All minor bugs are documented as bug reports.

26 -The documentation will follow test documentation standard IEEE829 5.Acceptance Testing -Test Scenarios prepared for all critical business flows. -Business value measured with test scenarios. -Test Scenarios prepared for usabilty tests and graphical look and feel tested. 6.Final Testing -Documentation complete -Automated Basic Flows are run -Automated IT is run -All NFR are tested Delivery includes: -AcceptanceTest plan -Test Scenarios -Test report, including fundamental metrics. -Final Report Customer Users test activities and Quality attributes Acceptance Testing Quality Attributes: All planned Test Scenarios have been executed completely. The system has no Critical or Major bugs. All minor bugs are documented as bug reports. 1-All documentation required is written and delivered 2-All basic business flows are run. 3-All integration flows are run 4-All NFR are tested 5-A End report is created

27 Buying Software from a Agile vendor 4-Put it in the contract Make sure they understand what they have to do. Make them understand that you will find their bugs. 1 time is a mistake, 2 times a big mistake and 3 times a habit. Make them understand that we will not do their work. ----Contract Issues Tests costs more Try to define a Fine in advance.

28 Buying Software from a Agile vendor The Three levels of delivery. Delivery of good software. Delivery of bad software. No delivery.

29 Buying Software from a Agile vendor The Three levels of delivery. Prioritised: 1-Delivery of good software. 2-No delivery. 3-Delivery of bad software.

30 Buying Software from a Agile vendor Running your UAT If the vendor is new for you, I suggest Pre UAT. After each delivery a UAT should be run. Agile UAT, sending your users (superusers) is OK as long as it really is a UAT, not a quest for bugs.

31 Buying Software from a Agile vendor It takes 2-5 years to teach a vendor what and how you want it. (But you must know first)

32 Buying Software from a Agile vendor If you still have problems. Buy in a Agile way. -Buy the requirement part first. -Then buy development. -Start over

33 Assessing improvements DDP DDP+

34 Assessing improvements Other important Metrics Requirement Coverage. (or coverage of what you decided to test) Time To Market, time from an idea until value for customer. Process Efficiency, % of the time a US is worked on. (3-7% common)

35 Assessing improvements Extremely important Metrics (1) Bugs in Production

36 Assessing improvements Extremely important Metrics (2) Happiness Index Values Look for fluctuation longer than 4 weeks. The first 10% of the employees that leave have 25% of the competence.

37 Questions?