Market Research Report Attitudes to Software Test Automation

Size: px

Start display at page:

Download "Market Research Report Attitudes to Software Test Automation"

Belinda Bradley
6 years ago
Views:

1 Market Research Report Attitudes to Software Test Automation An Infuse Consulting Research Paper Release 1.4 October 2012

2 PURPOSE STATEMENT This document is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Market Research Report Attitudes to Software Test Automation

3 Market Research Report Attitudes to Software Test Automation INTRODUCTION... 3 DEMOGRAPHICS... 3 REAL LIFE EXPERIENCE VERSUS PERCEPTIONS... 4 TYPE OF TEST AUTOMATION USED... 4 PLATFORMS USED... 5 REAL LIFE BENEFITS... 6 REAL LIFE DISAPPOINTMENTS... 7 PERCEIVED BENEFITS... 8 PERCEIVED DRAWBACKS... 8 INFUSE TEST-ON-DEMAND... 9 CONCLUSIONS ABOUT INFUSE CONSULTING GROUP Market Research Report Attitudes to Software Test Automation Page 2

4 91.7% of those who have used software test automation would do so again. INTRODUCTION Software test automation has evolved over recent decades from very basic record and playback through to scriptless and test inspection code scanning. Software test automation seems like a great idea to many especially those like Infuse Consulting who have developed sophisticated software and services to help businesses realise the benefits that automation can deliver. However, the investment made by many businesses to improve their software quality and to reduce testing effort has too often led to disappointment. Too often early generations of automation have been over-hyped and over-sold, such that those disappointed may have been deterred from trying newer and improved solutions that have evolved. So what is the reality? To find out Infuse has carried out research amongst business managers and test engineers. We contacted business people, some of whom have used software test automation of different generation, and some of whom have never been exposed to them. Some of these were senior managers from a non-technical background such as CEOs and CFOs. Some were more technically oriented (CIOs or IT managers) and some were software developers or test engineers. We wanted to find out what benefits or drawbacks were experienced by users of different styles of test automation. We also wanted to understand the perceptions from those with no previous direct involvement with automation to learn just how good or bad a job the industry has done over the years. The results were interesting. The biggest challenge for the industry would appear to be an educational one. There was a consistent drawback or perceived drawback across the board that many business managers do not understand the benefits they could derive from software test automation. A similar number believe they don t need automation, perhaps because they don t understand the benefits. It is reassuring that of all those who have used any form of automation, 91.7% stated they would use automation solutions again. Only 2.8% would not do so, with 5.6% being unsure. DEMOGRAPHICS Of the 87 respondents to the survey 59 had experienced test automation and 28 had not. There was a near even number of people from the business side of the organisation in each sector. None of the IT or development staff had experienced test automation, but all the quality assurance respondents had used test automation. A disappointingly large number of respondents did not provide details about their role or job title so we can draw no firm conclusions on the responses to the survey based on demographics Market Research Report Attitudes to Software Test Automation Page 3

REAL LIFE EXPERIENCE VERSUS PERCEPTIONS Of the 87 responders who completed the survey, 59 (67.8%) had direct experience with software test automation.

Some users have experienced more than one generation of solution, so were asked to provide answers for each type. The 28 responders (32.

5 REAL LIFE EXPERIENCE VERSUS PERCEPTIONS Of the 87 responders who completed the survey, 59 (67.8%) had direct experience with software test automation. They were asked to identify the benefits they had realised from automation by the type of solution they had used. Some users have experienced more than one generation of solution, so were asked to provide answers for each type. The 28 responders (32.2%) who have never been involved with software test automation solutions were asked to provide their PERCEPTIONS of what benefits or drawbacks they might expect to achieve. One responder stated that there was no such thing as test automation and gave no further feedback, so his responses are not included in the analysis below. TYPE OF TEST AUTOMATION USED Responders were asked to identify which type of software test automation solutions they had used. Some had experienced more than one type of solution. This was actually a difficult question for some people to answer accurately due to the different terminology used in the industry for some of the solution types. Test automation has evolved from record and playback (at the top of this chart) through to Test-on- Demand (towards the bottom). The profile of this was very much expected. Test automation technology has evolved from record and playback to scripting to data-driven scripts to action word / keyword to scriptless and Test-on-Demand. As the each new generation of solution has been in the market for fewer years so we would expect to have fewer responders for the newer technologies. It would appear that some of those who indicated they had used Test-on-Demand were confusing the Infuse Consulting service of that name with generic outsourcing of testing, hence the numbers here being just slightly higher than we might have expected. Market Research Report Attitudes to Software Test Automation Page 4

PLATFORMS USED It was interesting to learn that many of those with experience of software test automation had used more than one supplier s products.

6 PLATFORMS USED It was interesting to learn that many of those with experience of software test automation had used more than one supplier s products. HP s market-leadership position appears to be supported by this survey. Of those who indicated they had used other solutions the only solution mentioned more than once was for internally self-developed solutions. Market Research Report Attitudes to Software Test Automation Page 5

REAL LIFE BENEFITS It is reassuring that the survey indicates clearly that the newer generation solutions of test automation consistent scored higher scores on all the potential business benefits

7 REAL LIFE BENEFITS It is reassuring that the survey indicates clearly that the newer generation solutions of test automation consistent scored higher scores on all the potential business benefits than the earlier generation solutions. They also scored increasingly higher percentages of those who would recommend test automation. It is interesting though, that the percentage of those who would recommend their solution is lower than the 91% who would use test automation again. Perhaps they want to retain some business advantage for their own organization! The increasing business benefit and return on investment achieved with each new generation of solution clearly demonstrates that the industry has learnt from previous short-comings and has improved test automation effectiveness with each new generation. The overall benefits experienced improve with each new generation of test automation technology. Market Research Report Attitudes to Software Test Automation Page 6

REAL LIFE DISAPPOINTMENTS Of course, even with the improvements over the years, nothing is perfect. There are, therefore still some disappointments to be addressed by the industry.

8 REAL LIFE DISAPPOINTMENTS Of course, even with the improvements over the years, nothing is perfect. There are, therefore still some disappointments to be addressed by the industry. However, it is worth noting that these would appear to relate to increasing sophistication and capability of these tools making them more complex, thereby requiring more time to set up and more specialist staff. This may explain why businesses with large development environments and who wish to benefit from the newer technologies are looking to outsource their testing. It is worth noting that whilst all generations of solutions still raised some degrees of disappointment, the Test-on-Demand service showed the highest percentage of responders with no such concerns Market Research Report Attitudes to Software Test Automation Page 7

9 All respondents with no experience still perceive there are benefits from test automation. PERCEIVED BENEFITS The question on perceived benefits of test automation was asked only to those responders who stated they had no experience of these solutions. It will be of comfort to all suppliers of test automation solutions that all responders perceived some benefits. This does raise the question as to why those with experience of such tools and who would use them again believe many of their organisations management do not believe or understand the benefits. Only some 8%, though, believed there was a measurable Return on Investment. Perceived disadvantages of modern test automation relate to complexity of modern solutions. Perceived Benefits Response Percent Enables automation of largely manual and repetitive processes 72.0% Well suited to GUI testing 12.0% Enables testing to be executed earlier in the software delivery lifecycle whilst development still underway 24.0% Facilitates agile developments 20.0% Enables defects to be found earlier 40.0% Enables more thorough software testing 52.0% Allows more thorough regression testing on new releases 52.0% Enables test automation benefits to be gained on the first release of a new application 16.0% Allows the business to add more functionality sooner 12.0% Reduces time to market for new applications by more than 60% 4.0% Reduce cost of testing by more than 35% 16.0% Reduce effort of testing by more than 70% 4.0% Supports more frequent release-cycles 32.0% Allows you to test more often with the same or less effort 52.0% Improves the release delivery process 20.0% Good measurable Return on Investment 8.0% None of the above 0.0% Don't know 20.0% PERCEIVED DRAWBACKS Many of the perceived drawbacks are consistent with the actual disappointments expressed by those with experience of test automation, especially relating to complexity and the need for specialist staff. Again a lack of understanding of the potential business benefits and perceived lack of measurable ROI by senior management is evident. Market Research Report Attitudes to Software Test Automation Page 8

10 60% indicated interest in the Infuse Consulting Teston-Demand service. Perceived Drawbacks Response Percent Too time-consuming 20.0% Too complex 16.0% Not easily scalable 16.0% Not suited to embedded batch programmes 16.0% Software changes require complex changes to testing solution 16.0% Requires specialist technical staff to implement 32.0% Management of test scenarios is a huge chore 32.0% Cannot be used early in the Software Development Life Cycle 12.0% Not suitable for use on the 1st release 4.0% Not suitable for system under test 0.0% No measurable Return on Investment 16.0% The business does not understand the benefits of test automation 24.0% The business does not believe in test automation 0.0% There is a misalignment between the business and IT about test automation 8.0% Don't know 24.0% Other (please specify) 8.0% INFUSE TEST-ON-DEMAND Responders were asked to state how interested they would be in the next generation Test-on-Demand service from Infuse Consulting if Infuse could demonstrate a measurable return on investment. Some 64.3% indicated in varying degrees that they would be interested, with 35.7% being not at all likely to be interested. The reasons stated for those not interested were equally split between: Do not want or need such a service. Do not want to outsource or pay for testing. Satisfied with existing test solutions Don t really believe it can generate a Return on Investment This does suggest that there is education work to be done by the software test automation providers as some 10% of responders were totally closed to the idea that test automation can generate an ROI even if proven! However, when asked what would make them more likely to be interested in a Teston-Demand service only 3 responders were totally uninterested. The majority indicated that client testimonials, or peer recommendations or good ROI would actually make a positive difference to their attitude. Market Research Report Attitudes to Software Test Automation Page 9

11 The benefits of test automation have steadily improved Improved capability has led to more complexity, but outsourcing or Test-on- Demand services can enable you to benefit from the added sophistication these solutions offer. CONCLUSIONS Despite previous research carried out over the years indicating that the software test automation industry has a chequered success rate, the responders to this survey voted overwhelmingly (91.7%) in support of software test automation by confirming they would use automation again and are happy with their current solutions. The percentage of those experiencing business benefits increased with the newer technologies, particularly on key business advantage criteria such as enabling more functionality in each release, supporting more agile software development and finding defects sooner. If you have used an early test automation solution and were unconvinced of the benefits such that you have avoided them since; now might be a good time to reconsider and take a look at the newer solutions. If your business undertakes major software developments and you have never experienced test automation, have a negative perception of their capabilities and doubt their benefits to your organisation; this research suggests it is time to reconsider your position, keep an open mind and take a fresh look at how you too might benefit. The increased complexity that comes with enhanced capability of modern test automation tools is clearly creating a market for outsourcing of testing. Those companies who can not only offer the latest testing technologies but also offer testing as a quality service would appear to have an advantage over other suppliers of test automation solutions. There is also an opportunity for the leading test automation developers to both increase the sophistication of their solutions and make them easier for those end-users who prefer to manage their own testing and keep it in-house. The industry itself has some work to do to educate the large number of business managers that do not appreciate the benefits that software test automation can bring to their organisations. The industry also could do much better at educating existing users of test automation so they actually understand what type of solution they are using and how they work. In the current economic climate it is even more important than ever for businesses and non-profit organisations to reduce unnecessary costs, improve quality and improve business performance. This research suggests that software test automation can make a positive contribution in all these areas. Those businesses that have invested in automation gain competitive advantage over those that ignore it. Even if you are already convinced as to the effectiveness of test automation and are benefiting from your existing solution; it might be time to move onto a more modern technology to further improve your competitiveness and gain even greater returns for your business. Market Research Report Attitudes to Software Test Automation Page 10

12 ABOUT INFUSE CONSULTING GROUP Infuse Consulting (Infuse) is a leading provider of Quality Assurance and software testing services to organizations operating throughout Europe, the Middle East and South Asia. Infuse has presence in London - UK, Bucharest Romania and partner delivery in Pune India. The wide range of services from Infuse is designed to optimize the performance, quality and reliability of IT-enabled business systems, enabling them to be deployed with confidence. Our service offerings will also help you to minimize risk, reduce development costs, shorten time-to-market and work to predictable outcomes - factors essential to success in a competitive marketplace. Partnerships Our services and solutions are further complemented by strategic partnerships with a number of market leading companies, including Avnet, CSC, HP Software, Oracle, and Microsoft. We are a Gold partner of HP Software and Oracle. Infuse is a global HP Education Partner and besides being experts in the usage of the HP suite of tools, we contribute to HP s R&D program, notable achievements being development of the Infuse Inspectors. Infuse is also the preferred testing partner for Oracle Advanced Customer Services (ACS) UK. Our processes and methodologies are used as the underlying foundation for Oracle ACS Test Services which are also delivered by Infuse. Infuse is a key delivery partner company for the market defining Testing Maturity Model (TMMi) and is respected by our customers and partners in providing the best support, recommendations and delivery available. We have a solid track record of delivering within budget, with high quality and within timescales. Company Highlights 2002: Company founded 2004: Awarded ISO 9001 certification for quality managed systems and BTO service delivery 2004: Awarded partner status by Microsoft 2005: Announces strategic partnership with ITM for Middle East co-operation 2007: Awarded Silver Partner Status by HP Software 2008: Awarded Gold partner status by Oracle 2008: Saudi Arabia joint-venture Al-Kanar Arabia Ltd created in partnership with Prince Fahd Bin Migrin Al Saud and Nihilent Technologies Market Research Report Attitudes to Software Test Automation Page 11

13 2009: Oracle ACS announce Infuse as go-to testing partner for Oracle 2010: Major contracts won around Infuse Inspector Solution 2012: Romanian test factory opened 2012: Promoted to Gold Partner status by HP Market Research Report Attitudes to Software Test Automation Page 12

14 Attitudes to Software Test Automation Research Paper Reference: RP Att001 First issue October 2012 Author: David Hunt, Infuse Consulting Infuse Consulting 18 Soho Square London W1D 3QL United Kingdom Inquiries: Phone: +44 (0) Fax: +44 (0) Copyright 2012 Infuse Consulting. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. This document is for informational purposes only and may not be incorporated into a contract or agreement. Market Research Report Attitudes to Software Test Automation Page 13

Software Test-on-Demand - A Smarter Way

Software Test-on-Demand - A Smarter Way An Infuse Consulting White Paper October 2012 PURPOSE STATEMENT This document is intended for information purposes only, and may not be incorporated into any contract.