Regression Testing is dead. Long live Regression Testing!
The whole premise of DevOps revolves around the word continuous: Continuous Integration, Continuous Delivery, Continuous Testing and Continuous Improvement. But where does regression testing fit into this lifecycle?
As we all know, regression testing SAP systems is a vital function. The interconnected nature of SAP means a change to one business process can have unintended consequences in others. Uncovering these potential issues is vital to protecting production systems and ensuring the business is not disrupted when changes are applied. Even if you are using a waterfall delivery process (big, infrequent releases), only a very small proportion of the current processes the business relies on for day-to-day operation will actually be affected – at Basis Technologies, we call this the iceberg effect.
The small portion of an iceberg one sees above the waterline reflects changes to current business processes targeted for the next release. But, the vast majority of the iceberg is under the water – i.e. all of the processes that are not affected by the new functionality, but must still operate correctly after the release has been applied. In short, one tries to answer the question “will any part of these changes disrupt the way I do business today?” This question can be difficult to answer.
Regression testing traditionally takes a long time to perform, is difficult to automate, especially at high levels of coverage, and requires large amounts of effort and resource from the business and QA teams. And, let’s not forget, the best result one can ever achieve from a regression test is to find… Nothing. No wonder businesses think of it as non-value-add activity….
This brings me back to my question with DevOps - how can regression testing possibly be part of a continuous delivery process, as we can’t possibly run a regression test every day?
So regression testing is dead in the world of DevOps.
Well no, not quite. We just have to look at regression testing SAP systems in a new way. We need to shift regression testing from the end of the deployment process to the beginning (shift testing left) and automate it. Let me explain.
Historically, regression testing has been left to the end of the release cycle because, among other reasons, it is technically difficult to accomplish any earlier in the cycle. In fact, regression testing is often confused for user acceptance testing, where the focus should be on an 80/20 of key business processes and usability, instead of looking for introduced defects vs. the existing release. This is because a complete test of functionality often isn’t possible until the entire release -- and all its dependencies -- has been staged and properly configured in some kind of pre-prod environment.
This is wrong-headed. In practice, regression testing should begin as soon as the change is ready, and long before it has had the opportunity to ricochet through the landscape as a defect-in-waiting, eluding discovery until the big date for release to production. But because of the technical limitations of doing practical, large-scale regression testing before a complete integration assembly is available, as an industry we have accepted waiting until too late to do a good job.
What if one could decouple the dependencies between components of the stack and landscape? Assume the creation of the test harness, the necessary test data, and the tests themselves could be fully automated: with this solution in place, a full regression could run as frequently as desirable, even daily if ideal. Once run daily, a full regression includes testing every change developers have made the previous day -- the same techniques the non-SAP world has been using for years. Every morning developers see the results of the previous night’s test and fix any issues detected. This means a change does not even get to progression testing (SIT, UAT, etc.) until it has passed regression testing.
So, once the final progression testing has been completed, the change can safely be deployed to production whenever required -- it won’t introduce any regressions into production. Presto! Continuous delivery to SAP systems has been preserved, and regression testing has been completed!
The key here is fully automating regression testing in SAP systems. This is where Testimony, from Basis Technologies, comes in. Its revolutionary new concept of Robotic Test Automation fully automates regression testing, and without manually creating or maintaining test scripts or test data. Testimony learns how your production system is used by the business and turns this into a test harness automatically. Even better, it doesn’t just learn about how users interact with the system, it learns about all other processes that affect the system, such as interfaces and batch jobs, so the regression testing it performs covers the whole system and not just parts of it.
With Testimony, fully automated regression testing has become a reality and the ability to shift that testing left – to the beginning of the development process - ensures continuous delivery will not introduce regressions into the production systems. We really can adopt DevOps processes in SAP, whilst still maintaining the stability of our production systems – I bet the business would love that idea.
So, is regression testing dead? No. Long live regression testing!
Watch our webinar '6 steps to effective SAP testing that will keep your business safe' to learn more about Robotic Test Automation.
