Why Risk-based Testing in SAP is a bad idea
Pity today’s QA team. The volume of testing and validation asked of most quality organizations is far beyond team capacity. There are many well-documented reasons for this, including the rate of change in modern business, competitive pressures, regulatory requirements, and more. But, it doesn’t change the fact there’s more to test than can be tested.
The industry has come to accept this situation over time, investing in ways to reduce the potential business impact of not thoroughly testing each release. Enter Risk-based Testing, promising to determine what should be tested and what could be “safely” skipped, reducing the total test workload. The tools in this space generally use technical analysis to attempt to determine change, and how to prioritize testing attention, something not always well-connected between the process understanding the business works from, and the code the developers write.
In practice, approaches employing Risk-based analysis are a limited step forward. Risk-based Testing begins with the assumption it’s acceptable not to test some things because of a lower probability they will introduce defects. This assumption often incorrectly assumes code not directly modified will have no impact on system functionality. Risk-based Testing accepts as a fundamental premise lower test coverage, a higher risk of a defect making it through QA, and a greater chance of impact to the business by attempting to better prioritize the use of QA resources. Put simply, Risk-based Testing advocates skipping testing!
A more detailed technical analysis is required to fully understand the potential for risk using these approaches, but in general, the issue lies with the assumption dependencies and change can be fully evaluated from a technical analysis. While this is true for many kinds of change, such an approach often skips or is simply unable to detect whole categories of common dependencies and potential changes. Examples include master data impacts on untouched code logic, situations where references between objects are linked at runtime, integrations to external systems and many, many more.
In fact, even proponents of Risk-based Testing tend to recognize its limitations, especially for business systems. SAP themselves include the following disclaimer in the description of the Scope & Effort Analyzer - their own solution for Risk-based Testing:
“The analysis does not cover all project related efforts of a maintenance project. Furthermore the analysis result does explicitly not contain effort related to functional changes such as the activation of business functions. Data contained in the Scope and Effort Analyzer result report might have missing data or efforts compared to real maintenance project efforts.”
So, what can you do?
We know the combination of limited resources and manual testing won’t get the job done – that’s the reason behind the appeal of Risk-based Testing, after all. Test automation tooling is a must, but even comparing automation tools there’s a huge range of effectiveness and efficiency to be realized. An automation solution expecting a human user to spend time orchestrating (babysitting!) the execution of a test isn’t much better than manual testing; it just moves the work away from the subject matter or process expert to a technical user with a technical skillset. (Don’t choose that option; it won’t get you anywhere.)
When testing SAP, the better answer is to test everything and test every time, and this likely requires a multi-faceted approach. For regression testing within SAP, look at Robotic Test Automation (RTA). RTA solutions eliminate the need to maintain a test library and automatically provide the bulk of long-term test coverage, even as application use evolves. Without RTA many SAP users find they cannot automate enough; test library upkeep and maintenance ultimately consume resources otherwise used to expand coverage.
Use RTA frequently, and use it to move comprehensive regression testing to as soon as the developer thinks the code is ready - long before traditional manual or automated testing can be used because of their dependencies on a complete testing environment. This eliminates the overwhelming volume typical of regression requirements and will reduce the demand on QA to the testing of new features – a surmountable task. As a result, the QA team will have greater freedom to focus on the functionality changes the business requires from release to release, and test cycles will shorten dramatically.
For progressive testing (new features), demand from your automation vendor a scalable solution to automate test execution and reporting. It should be hands-off, lights-out, and run as frequently as practical, in many cases nightly as development integrates the components of a release. In fact, depending on your SAP environment, a pairing of manual testing for new features and Robotic Test Automation for the bulk of regression testing coverage may be ideal. What’s important here is to test everything every time: don’t accept any solution promoting less than that.
With the right solutions for SAP testing, there’s no need to rely on a risk-based approach, and no reason to accept the additional risk that comes with it. If testing time is constrained because of resources or release schedule, as is often the case, consider automation. With the solutions available today - including Robotic Test Automation - robust testing and full confidence in the functionality of each release is the new standard.
To learn more about why Risk-based Testing is a bad idea and the new approach that RTA provides, watch our webinar.
