The XBOSoft 2012 Scrum Testing Survey

Transcription

1 The XBOSoft 2012 Scrum Testing Survey

2 2 Introduction Test is dead. Chances are that if you attended a QA or testing conference in the past year or so, ou e heard this or something along this line from one of the speakers. Agile development teams supposedly have little or no need for testing, since testing will be done by the developers themselves. This, however, does not match our experience. What we see in the field working with agile teams all over the world is that testers have taken on a different (and, as we will argue later, perhaps an even more important) role in agile development processes. Traditionally testing was often viewed as the last line of defense for product quality. At the end of development the testers would run their tests against requirements to stop defects from entering the production environment, shielding users from buggy software. In agile development processes the objective is to continuously deliver finished and working portions of a product in short iterations of a few weeks. In this time frame the development team takes on all aspects of software development from requirements to coding to testing. This causes some people to say that dedicated testers are no longer needed in agile teams, because the developers should be able to do all tasks. Obviously there are a lot of changes with a different development process and it is entirely possible that a change causes roles previously important to be no longer needed. But we see that our testers who are working in agile teams are taking on more and different responsibilities than the traditional testing role requires. They have moved upstream and are now working with product owners and clients / users to specify requirements, building bridges between developers and users. This not only requires better communication skills, but also requires them to have more technical skills as well. We were interested to explore this further and decided to compare our experiences with those of other agile teams around the world. We chose to focus on Scrum for ease of comparison and because of its popularity as an agile development method. Scrum is a popular agile development method with specifically defined practices. Teams following Scrum have three core roles. A product owner, one person that represents the customers, the development team (that takes responsibility for all activities related to creating, documenting and testing the software) and a scrum master who is there to ensure that the development team can focus on building software. The multi-functional development team causes many to believe a dedicated test role is no longer required. Testing deals with the specific tasks of planning for, executing, and analyzing the results of, the tests. QA however is a much broader term that also includes development processes as well as other activities that an organization implements to achieve the desired levels of quality. What we currently see in the Scrum projects we are involved in is that there is high demand for testers to expand their roles both upstream and downstream. Upstream, in working more closely with developers, and downstream to act and think on behalf of the software users, causing the traditional testing role to expand into a QA role with more responsibilities. Let s see hat other orga izatio s e perie e. - The XBOSoft Team

3 3 From February to April 2012 XBOSoft conducted a survey on the state of testing in Scrum development teams with the objective to determine the state of testing in Scrum teams, to answer the question if they still use dedicated testers in their Scrum teams, and when they do, if they also experience an expansion of the traditional testing role into a broader QA role with more responsibilities. Key findings: We found that most people that participated in our survey have dedicated testing staff in their Scrum teams. But true to multi-functional teams even though they have staff dedicated to testing this does not mean this person (or persons) only participates in test activities. In the Scrum teams we surveyed, we saw testers participate in diverse roles ranging from documentation and configuration to programming. Usability testing is a popular activity in the scrum teams we surveyed whereby 20% of the respondents indicated that they conduct usability testing in every sprint. Even Scrum teams often feel they do not conduct sufficient testing.

4 4 Roles & Responsibilities We started by asking the Scrum team members if they had dedicated testers working in their team. Interestingly 94% of the respondents said they had at least one dedicated tester in their Scrum team. Do you have dedicated testers in your Scrum team? 6% No Yes 94% Figure 1: Do you have dedicated testers in your Scrum team? So despite the myth that with agile, developers do the testing and there is no clear test function, our survey results found that most of our respondents still have dedicated testers in their team. What this could possibly mean is that in Scrum teams, most teams find that developers are still not trained at seeing things from an end user viewpoint and there is value in having a separate test function. Additionally, it is also possible that Scrum teams have developers cranking out code, causing them not to have enough time for testing. Hence the agile model has not been fully implemented with too much on their plate to allow time for them to test. But how dedicated is dedicated in multi-functional Scrum teams? It is well known that developers in Scrum teams are supposed to take on different roles and responsibilities when compared to developers working in a team using a waterfall development model. Besides coding, developers in Scrum teams work on requirements and testing. But what about testers in a Scrum team. What responsibilities do they take on?

5 5 Do your testers also take responsibility for other tasks outside testing activities? 5% 39% 56% No testers in Scrum team No, they only do testing tasks Yes Figure 2: Do your testers also take responsibility for other tasks? A little less than half of the respondents answered that the testers in their team are also responsible for other tasks. The most common tasks are depicted in Figure 3: Configuration and Documentation both with 85 and 72 percent respectively. Tasks undertaken by testers outside testing activities Works in another role in the company Development and Story acceptance test Programming Management Database administration Documentation Configuration 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% % Figure 3: Tasks undertaken by testers outside testing activities

6 6 Note the testers that also take o progra i g tasks hi h ri gs to i d Elisa eth He dri kso s o ser atio he she first started to work as a tester in an agile team that she was the worst programmer in her team. ( True to the multi-functional set up of Scrum teams we see here that although dedicated testers are present in the teams they often have broader responsibilities than just testing. This reflects what we see at our clients: an expansion of the traditional testing role to broader quality assurance responsibilities. Using the waterfall development model, code is handed over the wall, and testers begin their work. In other words, testing comes at the end. With Scrum, small pieces of deliverable code with working functionality are delivered in fixed time frames (usually 2 weeks-called a sprint). Testing should occur within the sprint from beginning to end, starting with unit testing, and ending with acceptance or black box testing, all within the sprint. To discover if this really happens in Scrum, we asked our survey participants whether or not testing work started after all code was complete as shown in Figure 4. Programmers write small piece of code as quickly as possible and testing for that piece of code starts immediately Testing work starts once all coding work is finished 44% 56% Figure 4: For each sprint, how do you manage your testing? This is an interesting graph and one that needs further research. Because this graph shows that 44% companies that say they practice Scrum development still incorporate waterfall processes within their Scrum and do the testing at the end. This could indicate a common practice in the IT industry of wanting to follow the latest buzz words or could indicate that our respondents are using their own blend of Scrum.

7 7 Types of Testing Software testing consists of a broad range of activities. We were curious what types of testing our respondents conducted for each sprint. Functional, test case authoring Performance testing Automated regression testing Manual regression testing Type of testing for each sprint Function test Acceptance test Security testing Usability testing Exploratory testing Unit testing 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Figure 5: Type of testing for each sprint As expected in a survey among agile practitioners, unit testing is amongst the most predominant forms of testing. It is no surprise that manual regression testing is also common but what is very interesting is that more than one fifth of the respondents indicated that they conduct usability testing every sprint which could be caused by the more direct working relationship that Scrum teams have with their clients and end users of the software. This is something that will be investigated more in our future surveys. At XBOSoft generally we are proponents of automating UI tests and we often get called in to help our clients set up automation frameworks. So naturally we were curious what percentage of test scenarios / cases of our respondents are covered by automated tests.

8 8 What percentage of scenarios / cases is automated? 28% 22% 50% 0% 1% - 50% 51% - 99% Figure 6: What percentage of scenarios / cases is automated? In the agile projects that we work on, we often advise judiciously automating more than 50 percent of the manual test cases. With less automation, regression testing easily becomes a drag on larger projects where the team delivers in short iterations and test cases continue to accumulate. Enough Testing, Testing enough? For most applications, it is almost impossible to test everything. So choices have to be made as to what to test and not test. An important question for those working on software quality is always what the minimal amount of testing is that is required to reach the desired level of quality. Especially when working with the limited time frame of short iterations in Scrum this is not always easy. So we decided to ask if our Scrum participants if they thought they were able to test enough.

9 9 Do you think you conduct sufficient testing on your product? 11% 28% Always sufficient Insufficient in some sprints Always insufficient 61% Figure 7: Do you think you conduct sufficient testing on your product? Only 11 % of the respondents thought they always conducted sufficient testing. Indicating that there often is not enough test coverage in the Scrum teams we surveyed. For those participants that responded that they experienced insufficient testing, we proceeded to ask for more details regarding potential reasons as shown in Figure 8. What are the reasons for insufficient testing? 4% 8% 20% 40% Not enough time Not enough people Lack of testing skills No QA / Specialist Other 28% Figure 8: What are the reasons for insufficient testing?

10 10 Not enough time, not enough people and not having enough skills rank at the top for reasons of not doing enough testing. These items are not uncommon on any software development projects no matter what development method is used. Specific to agile and Scrum are the short time frames of the iterations and the objective to deliver a working portion of a product at the end of each iteration. The short time frame requires choices on what to focus on in order to obtain sufficient coverage. Conclusion Software testing using Scrum is a relative new concept in a recently developed software development process in still a relatively young IT industry. It is safe to assume that the last words have not yet been spoken on this topic. Where some say in Scrum teams the testing tasks should be taken up by the developers in our projects we see a broader and deeper role for testers in Scrum teams. But looking at the survey what can we discern so far? Our experiences as a Software QA and Test consulting company have been reinforced by what the respondents of our survey state. There is less need of a traditional testing role in Scrum teams, due to short iterations with fast delivery and deployment cycles there often is not enough time for a traditional testing role. This however does not mean that there is no need for dedicated software quality staff. We see testing in Scrum teams evolving into a broader QA role where the QA / test resources take ownership of the software quality and work with clients, users, and developers to ensure that the software meets the quality objectives. About XBOSoft XBOSoft is an American firm specializing in software quality assurance with branch offices in Amsterdam and Oslo. With a proven record of success working for Fortune 500 companies and specialized ISVs, XBOSoft has gained broad domain expertise with extended experience in finance and healthcare. Our commitment to software quality improvement is implemented through practical quality assurance processes combined with deep technical expertise. For more information, visit XBOSoft Inc. 640 Rocca Ave. South San Francisco, CA