I think I am lucky. Over the last decade or two, I have had the opportunity to use a good percentage of the testing solutions in the market today. Now, one might debate that as luck, but in my case the decisions had been made, and I got the benefit of watching and understanding the impact of tool selection on each organization I worked with. I found it highly educational.

What I learned is a set of key questions to help any organization improve the selection process of testing tools for SAP. I think of these questions in two general categories: business & requirements questions about the outcomes desired, and technical questions about getting there. I address each of these questions individually and explore how one might evaluate a solution in context.

Business questions:

• Do we need to automate test execution, or simply track it?
• Will the solution test (cover) enough?
• Can we afford what the solution will cost to own and support?  

Technical questions

• Will the solution scale?
• How will we maintain the test library?
• How will we solve for Test Data?
• Will the solution help with Root Cause Analysis and managing results?
• Does the solution support “shift-left”, or testing earlier?

Do we need automation?

Test automation offers huge advantages, but not for free, and for a smaller organizations may be difficult to prioritize. For example, some organizations may consider test automation’s requirements vs benefits excessive for their solution needs. Instead, these organizations use mostly manual testing and an ALM (Application Lifecycle Management) tool to track and report on testing practice and efficacy.

Larger organizations, or teams with high volume or regulatory requirements likely have little choice. Only automation provides the consistency, scale and thoroughness with each test run to satisfy the needs of these enterprises. 

If you are a smaller organization without a formal testing practice internally, and especially if you can sustain the occasional outage, not using test automation may be a practical choice. Use a simple test management and reporting solution (ALM) instead. The value of watching trend lines over time should not be underestimated: overall quality is often measured in aggregate, and changes such as development methodology can take a while to show impact through the entire system.

Coverage

This is the key consideration: how many business processes and individual activities make up the critical mass of your organization? What must *always* work, and thus, should always be tested? How many variations of core processes are essential?

Some automation solutions assist or even completely automate the discovery of business process, generating the test library automatically. For organizations with large testing requirements, these are significant advantages to consider in a solution.

I will caution against the allure of risk-based testing and other methodologies advocating for less testing. The appeal is obvious, but the reality is often just less coverage. Instead, evaluate what your business needs and select solutions as if these hopeful promises didn’t exist.

Can I afford this?

The hard question of any automation is affordability – and the costs come in both human capital and infrastructure. In general, a well-executed automation strategy and solution will help one scale to a higher level of test coverage than possible with manual testing. It also changes who does the work.

With manual testing, often individuals who otherwise perform productive business process are enlisted to verify functionality. With automation, a more technical skillset is responsible for solving the scripting test architecture challenges that come with those tools. In either case, detected errors are often reflected to a subject-matter expert team for evaluation.

Automation solutions typically require infrastructure investments. For example, a DevOps approach to testing SAP will require a test environment and deployment tooling suitable for the execution-at-scale of the test library. So, consider the technical requirements of using a selected solution as intended, and what acquisition and operational costs come with a given approach.

Scale

The whole purpose of test automation is to test more, in a shorter time period, and with less cost than a manual approach. This is a classic Quality-Time-Cost trade-off triangle but automation can improve all three at once. Doing so requires the solution and your organization to scale.

Does the testing solution perform test execution in an unattended fashion or does it require a desktop automation engineer to babysit each test execution? Will the solution scale up to the level of test execution required for enough coverage of my existing SAP functionality? Look for solutions with a fully automated capture & record as well as “lights out” testing playback capability to meet these goals.  

Test Library

Most test automation solutions have a level of script maintenance required for the solution to provide ongoing utility. The upkeep of these scripts can range from reasonable for the more elegant solutions to outright painful for others. Failure to consistently upgrade and maintain the test library will result in test scripts “breaking” – meaning they no longer work correctly and report false failures.

Over time, this will become a time-consuming maintenance task, limiting the ability of the automation solution to further increase coverage levels. Success is primarily one of talent and capacity: the test automation team’s technical ability and number of individuals dedicated to the task. Finding and retaining talent familiar with the selected solution, especially for desktop-user-interface based solutions, is essential.

Test Data

Test data are the values and inputs used during test execution and may be closely tied to existing information in the system. For example, to properly test an order-to-cash process, one may need the data for a customer, sales channel, order type and materials, and other details. The order then requires underlying materials inventory, an invoice account, and so on. These must be in place for an order-to-cash test script to run correctly, and ironically, just running the test often exhausts this supply of data from the QA system.

Managing test data can be one of the more challenging aspects of success with automation. The best testing solutions significantly reduce or even eliminate this problem with various technical and methodological approaches. Using a production system copy approach is one way of addressing this challenge unique to SAP environments.

Root Cause

Once a defect has been identified, what process validates the defect, then remediates the issue if confirmed? Triage may involve a range of individuals, from a functional expert to evaluate the potential defect, to a developer to correct the code. The best solutions coordinate the team and understand both sides of the system: the needs of audit and reporting to the process, the failure from the user experience perspective, and even what technical changes to objects and programs are likely the cause.

Automation in this area becomes especially important as test size and frequency increase. False alarms and unnecessary triage labor can overwhelm the best of good intentions when testing tempo increases. Look for solutions allowing the team to test more and manage test outcomes more efficiently. 

Shift-Left

The concept of shift-left is a founding principal to the application of DevOps in regular release testing. In short, it means testing as soon as possible with the benefits of getting your bad news sooner rather than later. The challenge for SAP testers has been one of having a functioning environment to test in: often, it means a fully integrated QA environment with external systems and other complexities of the production environment. This means basic regression testing must wait until very late in the test cycle when this total assembly is complete. 

Automation enabling regression testing prior to this ‘total assembly’ stage can deliver significant value. In SAP practice, this allows a development team to know a change does not disrupt existing business functionality before the transports are released to QA. Earlier detection of defects saves QA and Development resources and prevents unproductive rework cycles in the release process.

A portion of this challenge is addressed outside of SAP with a technology called Service Virtualization. SV allows components of an otherwise larger infrastructure to operate independently for the purpose of early testing by removing (emulating) external dependencies. Solutions for SAP bringing an analog of this functionality help address “waiting until pre-prod” for regression testing.

A comprehensive testing solution for SAP should contribute to shift left in a meaningful way, either through direct functionality or integrations with external tools. 

Final Thoughts

Hopefully, these questions help your approach to selecting an automation solution for testing SAP. Not all may apply to your organization, but it should help round out your requirements and shopping list for solutions in the space.

If you have a favorite point of inquiry on this topic, please share! I’m always open to new recommendations to help organizations make the best testing decisions.