SAP Testing Tools Comparison: Key Considerations Tricentis All rights reserved.

Transcription

1 SAP Testing Tools Comparison: Key Considerations 2017 Tricentis All rights reserved.

2 Companies relying on SAP must continually test their systems to avoid business disruptions stemming from SAP upgrades, service packs, customizations, and enhancement packages. Failure to detect or address a change could bring critical business processes to a standstill. Yet, much of the testing responsibility falls on key users, who are expected to validate transactions alongside all their other duties. At the same time, the complexity of what needs to be tested is increasing, with end-to-end business processes commonly extending beyond SAP and into APIs, web UIs, mobile interfaces, custom back-end systems, packaged applications, and more. How can you effectively manage change in light of these challenges? Leading enterprises have found that SAP test automation tools are critical for detecting and mitigating impacts to core end-to-end business transactions. To help you choose the SAP testing tool that best suits your organization s specific needs, this paper outlines some key points to consider as you evaluate and compare tools. 2

3 Ease of SAP Automation With inadequate or laborious SAP automation technology, your test automation initiative will be an uphill battle from the start. It will also likely be a losing battle if it s easier to manually test certain elements than to create and maintain automated tests for them. Consider: How much manual effort is actually involved in defining the automated tests? In updating them? What SAP elements can be automated out of the box? For example, does the tool provide out-of-the-box support for: Unlimited navigation in SAP tables via business readable content identification How are dynamic SAP Control-IDs handled? In what situations is coding required? How frequently will the associated coding need to be updated? Can tests be automatically generated from logs? Recording user behavior? Drag and drop from a library of options? Is the interface intuitive? How long does it take end users to get up to speed and recognize the tool s value? How much manual effort is required to reuse test blocks (e.g., for an SAP login process)? Verification of values in result tables (technically: label containers) Status checks in SAP tables Navigation in SAP trees via business instruction Creation and verification of IDocs The various SAP Fiori elements associated with your processes 3

4 Autonomous SAP Testing Automated generation of SAP tests from production data has the potential to make business process testing significantly faster and easier. However, it s important to consider exactly what steps are automated and how intelligently. Consider: How does the tool generate a list of processes to test? Does that list have an item for each and every transaction detected, or does it analyze and correlate the data to deliver a master superset of transactions? Does it prioritize the transactions and help you determine which are most important to test? Does it actually generate tests that cover the most critical transactions? How much manual intervention is required to create these automated tests? How are the tests updated as usage patterns change? How rapidly does the resulting test suite execute? 4

5 Technology Support Beyond SAP Without easy, intuitive SAP automation, your business process testing initiative will be dead on arrival. However, the ability to extend beyond SAP and cover every technology involved in your end-to-end transactions can make the difference between fragmented spurts of automation and a comprehensive, trusted test automation strategy. Consider: If your core business transactions involve technologies such as Java, mobile front-ends, SOAP or RESTful APIs, etc., how easily and effectively can the tool test those elements? Can the tool automatically test the integrity of any BI data that your business relies on? Can a business user feasibly automate end-to-end tests that include non-sap technologies such as mobile, APIs, etc.? How much additional training would be required? How quickly does the vendor introduce support for new technologies? Can values from one type of test be used to parameterize other types? For example, can the result of an SAP field be used to populate an API test? Can you use a central test data management and generation interface for all your tests, or do you need to have separate solutions for SAP and non-sap elements? 5

6 Risk-Based Testing Business analysts are burdened with testing in addition to all their other duties. Resources for testing are limited, so it s critical to understand and focus testing efforts on the top business risks. Consider: Does the tool help you prioritize testing based on the risk associated with your requirements/user stories? Does the tool help you determine what tests and test data are required to cover your top business risks? Does it tell you what level of business risk coverage your tests are achieving? Does it report test results in terms of your requirements and risks? Does it help you determine the minimal number of tests needed to exercise all application logic? 6

7 Impact Analysis The increasing pace and scope of SAP updates requires comprehensive, near-continuous testing to identify and resolve potential impacts. You not only need to know what changed; you also need to ensure that your test suite is efficient. Otherwise, you might waste time continuously updating tests that do not add value. Consider: Does the tool help you identify and update tests impacted by changed SAP transports including end-to-end test cases that extend beyond SAP? Does it help you generate dynamic TBOMs? Does it tie the results of the impact analysis to risk-weighted requirements (from Solution Manager) to help you prioritize updating? Does it help you continually optimize test suites so you don t waste time updating no value added tests each time a transport changes? 7

8 Test Maintenance and Optimization Many organizations create automated tests, but few achieve sustainable test automation rates. Often, false positives, timeouts, and incomplete tests erode trust in the results. Also, the test suite grows too unwieldy to update to the point where it seems easier to test manually than update everything that s out of synch. Consider: How easy is it to update tests if an element is added, removed, renamed, etc.? How easy is it to update test data used across the transaction (SAP and non-sap elements)? Does the tool help you reduce test case redundancy to prevent bloated, difficult-to maintain test suites? Does the tool ever create duplicate modules when tests are created? If so, how are they resolved? Can you execute tests even if dependent components are unavailable or in an unstable state? 8

9 Readiness for Agile and DevOps Studies report that 81% of organizations are adopting DevOps and 91% are adopting Agile. However, you d have to search long and hard to find an enterprise that doesn t have scaling Agile and adopting DevOps on its radar. It s not a matter of if testing will need to accommodate DevOps and Agile, but when. Consider: Can you integrate the tool into a best of breed DevOps pipeline with a variety of CI/CD tools? Does it offer bidirectional synchronization of requirements, issues, and tests with popular tools such as JIRA, Rally, GitHub, etc.? What capabilities does it offer that help you ensure rapid, successful execution of unattended tests? Does it offer service virtualization to ensure that end-to-end tests can be executed during CI without dependency issues causing timeouts, incomplete tests, false positives, and/or inaccurate results? Does it help you prioritize a condensed set of smoke tests suitable for CI as well as a larger set of nightly tests that provide broader risk coverage? Are there ways to achieve the desired test execution speed (e.g., with distributed execution)? Does it enable shift left defect detection techniques such as exploratory testing? Does it allow you start testing even when dependent components (e.g., third-party applications, APIs, etc.) are not yet completed or are evolving in parallel? 9

10 Analyst Evaluations In addition to your own research, you might want to consider what analysts have to say. Each year, Gartner evaluates the top tools that met their criteria for enterprise-grade test automation tools. The evaluation involves inquiries with Gartner clients, surveys of tool users, vendor responses to Gartner questions, and product demonstrations. This year, the results were as follows: For the detailed analysis, including the pros and cons of each tool, you can download the latest report from the Tricentis web site. You can also review a synopsis of the previous year s rankings on TechBeacon. 10

11 Take the fastest and easiest path to automated business process testing Learn about Tricentis SAP Testing zero-touch, scriptless testing, industry-leading SAP test automation & more Stay in touch Locations Contact