ag ile / ajel/ CONTENTS Introduction What is Agile? Agile Testing Survey Survey Summary The Future of Agile and Agile Testing

Transcription

1 ALL ABOUT AGILE TESTING SURVEY REPORT 2015

2 CONTENTS Introduction What is Agile? Agile Testing Survey Survey Summary The Future of Agile and Agile Testing ag ile / ajel/ 1. Able to move quickly and easily 2. Able to think and understand quickly 3. Relating to or denoting a method of project management, used especially for software development, that is characterized by the division of tasks into short phases of work and frequent reassessment and adaptation of plans [Def. 1, 2, 3] In Oxford Dictionaries, Retrieved September 12, 2015 from CONTACT US AT SERVICES@ 1

3 INTRODUCTION From the inception of lightweight software development methods in the mid-1990 s, to the realization of the Agile Manifesto by a group of seventeen software engineers in , agile has become an ever more common software development methodology. Despite its detractors, agile has continued to grow in popularity among development teams, and today challenges waterfall as a development practice. Some developers are finding a healthy mix of the two methods affords a best of all worlds approach. 2 Since the success of agile is dependent on nimble communication and the collaboration and commitment of all stakeholders, others are finding it s not the best fit for every organization. Love it, like it or hate it, agile has become entrenched in the software development scene. XBOSoft has worked with an increasing number of clients who have migrated to an agile methodology over the past number of years. In response to its rapid growth and the debate that often surrounds its implementation and principles, we undertook a survey to uncover answers to a variety of questions. The following is a summary of those survey results that provides insights into the state of agile, agile testing methods and techniques, and the use of agile in business. What works, what doesn t and what lies ahead. The who s, the what s, and the how s The survey was run from January 2015 May 2015, promoted to the XBOSoft audience of software testing and development industry professionals via digital and traditional marketing channels. 1 Agile Manifesto Organization CONTACT US AT SERVICES@ 2

4 MANIFESTO FOR AGILE SOFTWARE DEVELOPMENT We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:» Individuals and interactions over processes and tools» Working software over comprehensive documentation» Customer collaboration over contract negotiation» Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. BUT FIRST... WHAT IS AGILE? Agile is typically defined as any project management approach in which effort is made to work jointly as a team respecting principles of collaboration, flexibility, simplicity, transparency and responsiveness to feedback throughout the process of developing a new program or product. Agile calls for efficiency in maximizing the amount of work not done, 3 balancing objectives of velocity and quality undertaking quality assurance testing procedures that conserve energy, effort and resources while mitigating risk. The method requires attention to detail; responsiveness throughout the development process; thorough, flexible testing; and a clear picture of the final product. Adaptability and rapid response are critical characteristics of an agile project. Roles are fluid, with a greater focus on skills and collaboration. Agile is a state of mind Optimal agile results come from remaining adaptable and relatable to an organization, examining current processes and assessing problem areas, goals and objectives. It s not about strict adherence to agile principles and methods, but rather remaining open, resourceful and flexible enough to use them as guidelines a frame of reference. Successful teams think agile towards a clear end goal. 3 Principles behind the Agile Manifesto CONTACT US AT SERVICES@ 3

5 Benefits & objectives QUALITY: One of the main objectives of agile is to not only provide working software, but software free from defects. It s important to test for defects that affect the customer or end user prior to production. Short iterations provide opportunity for timely and immediate testing, fixes, and response. VELOCITY: Agile development aims to streamline the software production process, improving investment of time and delivery for those functions and features that are most valuable to the stakeholders. Reliable metrics that track what s working and what s not to allow for nimble response time and feedback is critical in the agile environment. The agile argument Those who argue against agile as an increasingly popular development methodology are often passionate in their appeal. The counter argument cites the focus on feel good team work and lack of specificity in production for less than stellar results experienced by some organizations. And emphasis on velocity over quality speed over improvement means product may be quick to market, but at what long term cost and sacrifices in quality? As teams race toward agile implementation, testing is often misplaced as a critical activity. Skills over roles, interactions over processes, and working software over documentation are interpreted as agile principles that don t stress testing or don t require a specific testing function on the team. As a result, many organizations that convert from waterfall struggle over what to do with testing. ( What do we do with our testers? How can they work more closely with developers? ) This survey was conducted, in part, to investigate how organizations are solving these issues today and what the trends are. CONTACT US AT SERVICES@ 4

6 OUR AGILE TESTING SURVEY * Respondents were able to select more than one answer 1. Why do you use agile? * Accelerate release of product Changing responsibilities Increase productivity Project visibility Reduce risk Simplify dev and test process Reduce cost Manage distributed teams Enhance software quality 68% Accelerate release of product, 65% Increase productivity, 55% Enhance software quality We began by asking respondents the key question why they implement agile. According to survey response, businesses are focused on speed to market and productivity, with accelerated release of product the overriding objective, followed closely by increased productivity. The nimbleness the methodology provides and enhanced software quality also scored high as responses. CONTACT US AT SERVICES@ 5

7 2. What actual improvements have you achieved from agile? * Accelerate release of product Increase productivity Project visibility Reduce risk Simplify dev and test process Reduce cost Manage distributed teams Enhance software quality 63% Accelerate release of produce, 51% Increase productivity, 47% Enhance software quality Agile implementation produces outcomes for survey respondents that closely mirror objectives expressed in Question #1. Over 60% experience accelerated release of product, and more than half see increased productivity. CONTACT US AT SERVICES@ 6

8 3. What metrics do you use to measure improvement for agile?* Velocity Defects in production Other (please specify collected metrics) 60% Defects in production, 54% Velocity, 38% Other Defects in production is the most commonly used metric in agile, followed by velocity. Other metrics mentioned by survey respondents included:» Tickets complete» Happy customers, increased collaboration, skill sharing and development» Working mood» Bugs reported by end user» Test coverage» Sprint capacity, productivity, backlog depletion rate» Cost of each feature vs. revenue earned These metrics reiterate some of those mentioned in XBOSoft white papers (including Agile Quality Metrics and Agile Velocity Metrics) where defect metrics classified as resulting metrics and others such as test coverage, working mood and overtime, if measured, can be used to predict defects. Among the survey replies was the response that agile is just a fad indicative of the debate that continues despite the method s growth in popularity. CONTACT US AT SERVICES@ 7

9 4. What kind of agile methodology do you use? Scrum Kanban Extreme Programming (XP) Other (please specify) 78% Scrum, 15% Other, 5% Extreme Programming Out of the short list response to the survey question What kind of agile methodology do you use, Scrum was the obvious frontrunner. No one method is best each can suit a particular situation and context. While Scrum is more popular, it doesn't mean it s for everyone. Kanban has its own merits in flexibility and they are not mutually exclusive. It s interesting to note that two respondents commented all of these and more and we focus on sprints. Additionally, other was more widely used than both Extreme and Kanban combined, indicating that many develop an agile hybrid to suit their own particular needs. CONTACT US AT SERVICES@ 8

10 5. What agile techniques do you use? Daily stand-up Iteration planning Test automation Retrospectives Unit testing TDD 79% Iteration planning, 75% Daily stand-up, 69% Test automation Teams employ both iteration planning and daily stand-up as agile techniques. 69% of respondents use test automation when and where appropriate. Other respondents mention:» Automated+crowd» Automation where appropriate» Behavior Driven Development, no estimates» Iteration Demo» Continuous integration/test It s noteworthy that from a testing perspective, many organizations employ automation and unit testing the best way to keep up with quick iterations. On the other hand, it might be thought that TDD (Test-driven Development) should be more prevalent. However, to have TDD, requirements are needed to set up tests, and in agile requirements are often times too fluid. What is perhaps most interesting, is that only 54% use retrospectives. Although continuous improvement is a primary agile principle, it appears that many teams are overlooking this critical activity as a means to improve. CONTACT US AT SERVICES@ 9

11 6. How do you manage testing for each sprint?* Programmers write piece of code quickly and testing follows immediately Testing work starts once all code is finished for that sprint We log defects as we go during the sprint 51% We log defects as we go during the sprint, 45% Programmers write piece of code quickly and testing follows immediately, 35% Testing work starts once all code is finished for that sprint Logging of defects as a popular form of sprint management parallels the most commonly used agile metric among respondents (See Question #3). However, survey results also indicate that half of participants do not log defects meaning defects then cannot be used to provide information about increasing or decreasing quality or analyzed for root cause analysis. Almost half of respondents reported that programmers write code quickly, followed immediately by testing. Response from 35% saying that testing work starts once all code is finished suggests that although these organizations may be implementing agile in some ways, they are also carrying over some waterfall habits. Comments included that team members work together all the way through development or that work starts in parallel and that iterative testing happens at the story level. CONTACT US AT SERVICES@ 10

12 7. What types of testing do you have for each sprint?* Functional Test case authorizing for functional Acceptance test Security testing Usability User experience Performance testing Automated regression testing Exploratory testing Manual regression testing Unit testing 90% Functional, 61% Unit testing, 53% Acceptance testing A full 90% report the use of functional testing to verify a sprint. Unit and acceptance testing methods are conducted by just over half the survey participants. According to other comments, some teams undertake performance testing based on developments in the product rather than with each sprint, while many express a struggle to allocate testing time within sprint boundaries. CONTACT US AT SERVICES@ 11

13 8. What percentage of scenarios/cases are automated? 0% 1-50% % 55% (1-50%), 31% (51-100%), 15% (0%) Best use of automation in agile remains a bit of a debate. 55% of survey respondents use test automation in 1-50% of agile project cases. Approximately a third use automation in % of agile environments. 15% don t make any use of automation. CONTACT US AT SERVICES@ 12

14 9. Developers - To test your solution, teams following a Test-driven Development approach also: Have testers embedded in team Perform end of lifecycle testing Have parallel independent testing Other (please specify) 60% Had testers embedded in team, 18% Had parallel independent testing, 14% Other According to survey results, embedding testers in agile teams is the preferred testing method when following a Test-driven Development process. It's not surprising that "Performed end of lifecycle testing" produced the lowest response since this question was targeted toward TDD teams. However, while TDD will ensure each function operates correctly according to the test written prior to development, what if the test and the function are implemented incorrectly? This kind of defect, while not a defect according to passing the initial test, would be significant to the end user. CONTACT US AT SERVICES@ 13

15 10. Do you think you conduct sufficient testing on your product? Always sufficient Insufficient in some sprints Always insufficient 65% Insufficient in some sprints, 31% Always sufficient, 4% Always insufficient While many respondents reported positive results when it came to improvements achieved with the implementation of agile (see Question 2), a good number (65%) also feel the level of product testing in some sprints is insufficient. Only 31% feel that testing is always sufficient. Testing is never complete, whatever the methodology being used. The question is how much risk can be accepted and how forgiving end users are. Different levels of risk can be accepted depending on domain. Gaming, accounting and healthcare applications would have disparate risk profiles due to the nature of the consequences resulting from a defect. CONTACT US AT SERVICES@ 14

16 11. What are the reasons for insufficient testing?** Not enough time Not enough people Lack of testing skills No QA/Specialist No test automation Other (please specify) 76% Not enough time, 38% Not enough people, 30% No test automation Lack of time was the most noted reason for insufficient testing, followed by shortage of people. Other testing challenges mentioned included:» Incorrect sizing of work» Scrum team not recognizing/owning testing as a team activity» Lack of test automation, TDD not followed, Design Death» Missing automated test cases» Lack of product knowledge» Stakeholder involvement of user stories Inadequate time and people resources are nothing new to testing. However, they are exaggerated with agile. Agile is typically best implemented with small, multi-disciplinary teams of nine or less. Skills, as the Manifesto states, are deemed as more important than roles, with team members encouraged to step in and help out where and as needed. If an agile team lacks people specialized in critical testing functions (i.e. automation), or includes business analysts who can test but systematically don't do so due to lack of training, then the result is insufficient testing. Incorrect sizing of work is another reason that testing doesn t get done. Complexity of features and their implementation is often underestimated, resulting in last minute testing or a feature simply not being completed. CONTACT US AT SERVICES@ 15

17 12. What are the most challenging aspects of testing when implementing agile?* Adopting Test-driven Development approach Validating non-functional requirements Getting stakeholders/clients involved with testing User interface testing Adopting new agile testing tools Migrating existing testing and quality professionals to agile Using our existing testing tools to support agile dev Remaining regulatory complaint Other (please specify) 37% Using our existing testing tools to support agile dev, 37% Getting stakeholders/clients involved with testing, 34% Adopting Test-driven Development approach The challenges in testing related to agile adoption were somewhat evenly split out of a range of possible answers. Having to make use of existing tools to support agile implementation and getting all stakeholders and clients involved were just slightly ahead of adopting Test-driven Development suggesting that the overriding challenges are mindset and full team participation. CONTACT US AT SERVICES@ 16

18 13. What are the leading causes of a failed agile project?* None failed Organization principles don t match agile values Still many 3rd party pressures requiring waterfall processes Lack of experience Lack of management support Product owner not involved enough New to agile Lack of training Other (please specify) 34% Organization principles don t match agile values, 34% Still many 3rd party pressures requiring waterfall processes, 29% Product owner not involved enough Lack of full team or organizational support is the chief reason for failure. In some cases, agile is, in simple terms, not a good organizational match due to company culture, goals or principles. Out of the reasons given, some suggest that agile is a dysfunctional method with time and inability of data and team resources to keep up of prime concern. Because agile is a relatively new software development methodology, there are no real long-term studies or data points that prove it is more, better, or worse. In our experience, culture and organizational structure are the main inhibitors to successful agile implementation. CONTACT US AT SERVICES@ 17

19 14. Would you be interested in an Agile Testing Certification? Yes No 49% Yes, 51% No Given that agile is still a rather new method, it's not surprising that so many respondents said No to interest in an Agile Testing Certification. As noted earlier, agile is a mindset. Although there are several certifications, you must first think and be agile before you can do agile. Additionally, it may be that with the number of different kinds of testing certifications available, the market is already saturated. 15. Do you outsource the testing in your agile project? Yes No 49% Yes, 51% No When considered along with responses to the next question on working across the globe with geographically dispersed teams, it appears that although organizations are willing to simulate co-location on an intra-company basis, they don t seem to be outsourcing. It could be that integration of the test team with development is seen as too difficult for outsourcing, whereas there s less choice for intra-company teams. CONTACT US AT SERVICES@ 18

20 16. Do you work with remote agile teams across the world? Yes No 62% Yes, 38% No What is noteworthy about the survey result is that although agile principles value co-location to facilitate better collaboration, the reality is that many companies have multiple locations. This forces companies to work in different locations, thus simulating co-location via tools, video-conferencing and other methods. CONTACT US AT 19

21 17. What tools do you use with agile development and testing?* Version One Atlassian/Jira Microsoft TFS Bugzilla IBM Rational Team Concert HP Quality Center IBM ClearCase In-house tools Other (please specify) 63% Atlassian/Jira, 25% Bugzilla, 23% HP Quality Center In addition to the provided answers, other responses from survey participants included:» Whiteboards» YouTrack/TestRail» Varies from team to team» Test Link» Spira» Jenkins for automated test execution» Rally» Spiratest, Selenium, Ranorex, Junit, Jmeter» Jenkins, Micro Focus SilkTest» Agile designer CONTACT US AT 20

22 SURVEY SUMMARY What does the survey tell us? Over 60% of clients use agile with the objective of accelerating release of product and improved productivity. Both are proven achievable goals, with 63% reporting speed to market and just over half of respondents seeing increased productivity. While defects in production followed by velocity are commonly used agile metrics, there lacks a consistent set of metrics being used by organizations to measure agile productivity and quality in the beginning or midst of the process rather than at the end. According to survey results, all stakeholders must be fully on board, supportive and collaborative for the successful implementation of agile. Insufficient testing is a noted concern, due most often to lack of time and people resources. Iteration planning, daily planning and test automation rank high as types of agile techniques used by respondents. Yet, when combined with expressed challenges with agile implementation, such as stakeholder buy-in or product owner involvement, it can be ascertained that companies, while going through the motions of what they learned in training (i.e. daily stand-ups) may not, in actuality, be agile, but merely doing agile. And as previously stated, agile is a state of mind a way of being not a doing. Agile as a method, however, is recommended by 87% of those surveyed. THE FUTURE OF AGILE AND AGILE TESTING Predicting the future of any technology, methodology or advancement would be injudicious. However, agile continues to grow as a viable software development method in a rapidly evolving IT landscape. And its founding principles may be what sustain it. At its core, agile is a method developed to respond to change, innovation, streamlined procedures and cost-saving measures those same characteristics critical to the success of any business or venture in today s highly competitive environment. Perhaps, too, agile speaks to a desire, at a human level, to collaborate and work more closely together in small teams. The Agile Manifesto was conceived almost 15 years ago. While agile may continue to evolve as a methodology, it appears to be here to stay for the foreseeable future. A few of the issues surrounding its evolution include:» Toolsets that don t foster collaboration between different departments including product management, development, testing and customer support with the end users.» Mindsets that rigidly define Agile rather than use agile as a framework to adapt.» Shifts in corporate culture that are needed to implement agile in larger organizations. Given response to the methodology, it seems certain that users will rise to the occasion and solve these and other challenges over time through iterating and evolving, the agile way. XBOSoft thanks those who participated in the survey. CONTACT US AT SERVICES@ 21

23 ABOUT XBOSOFT XBOSoft is a software testing and quality assurance company. Our quality process assessments and test methodologies speed products to market and improve clients software quality and performance throughout the lifecycle. With QA services in web, mobile and desktop applications, we offer broad domain experience with extended expertise in healthcare and finance. CONTACT XBOSOFT WEB: xbosoft.com services@xbosoft.com