The most common testing problem is getting consistent results from a suite of automated UI tests(often referred as regression tests). This issue usually arises when QAs keep adding new UI tests to the regression suite without any proper testing strategy.
While this may seem like a good idea initially, it soon becomes a major issue for the team. Growing regression tests become unreliable, inconsistent, and slow in execution.
This leads to various other problems that affect the business, with prolonged release cycles being one of the most common consequences. Conversely, if the team decides to shorten the testing cycle, they risk compromising quality and may receive high number of defects.
If QAs decides to address the issue of inconsistency and stop adding new tests, they start loosing test coverage in few areas. It introduces another problem as they have to rely on manual effort to test the areas which has no automated test coverage.
The scenario described above is a classic example of how automated testing done without a strategy many introduce many problems. Many teams recognise this problems and attempt to address it, but often they fail to fix it properly.
[Read: How QAs can contribute to application code review]
Common attempts to solve the problem
#1 – No worries, Let’s add more QAs to the team
Let’s fix this problem by hiring more QAs. A few of them can handle inconsistent tests, others can focus on tuning them to run in parallel, and if there is still few available, they can add new tests.
But does it actually solve the problem?
No, it doesn’t. While there may be some improvements, such as faster execution or increased test coverage, the growing volume of regression tests will always have some repercussions.
#2 – Change the Tool
Ok. So Selenium WebDriver is the main culprit, as it is an old and not super fast library compare to libraries like Cypress, PlayWright. Also, team did not follow right coding practices.
Let’s change the tool and framework, this time we will be following clean code practices, making the tests modular and will ensure that the tests run in parallel.
This is another way to solve this problem.
But the question remains same, does it really solve the problem?
Yes, it will solve a few things. But changing the tool or following the right coding practices can never eradicate the problems completely.
Growing tests can still be challenging to retain it’s consistency. They may run faster but if tests are poorly strategised then the problem can re-occur.
Strategic Changes to Solve the Problem
Redefining the regression
Regression testing ensures that recent code changes have not broken any of the existing functionality.
But is it necessary to test everything through browser only? Is there any other better way to test?
Yes, APIs are great alternative to perform regression testing. Teams aiming for an effective and efficient testing should move their majority of tests at API level. APIs are great alternative of UI tests, it runs faster as well produces consistent result.
Some might argue that API tests alone don’t ensure the UI works as expected. Since users interact with the UI, it’s important to test it.
I would respond them with, having API tests do not mean ignoring the UI. Instead, you should balance UI and API tests to ensure all cases are covered. Here is a table showing how tests can be distributed between UI and API to achieve full coverage.
| Test Case | API Coverage | UI Coverage |
|---|---|---|
| Verify discount is successfully applied to cart | May Be | Yes |
| Verify failed discount error message “Promo Expired”, when entered promotional code is invalid | May Be | Yes |
| Verify failed discount error message “User is not eligible”, when entered promotional code is not eligible for logged in user | Yes | No |
| Verify failed discount error message “Promo is not applicable to selected item”, when entered promotional code is not eligible for Apple products | Yes | No |
Rebalance the pyramid
Another common reason for inconsistent tests is the anti-pyramid approach of testing. The team does not add test coverage at the lower levels and relies only on UI tests for all possible cases.
As mentioned above that a majority of UI tests should be replaced by APIs.
Additionally, the team should add tests at the lower levels of the pyramid. They should break the end-to-end business features into smaller scenarios and test them at the component level. Component-level testing isolates dependencies and allows testing of scenarios that are hard to achieve when various services are integrated. Component level tests are fast, reliable and easy to maintain as these tests sit close to the application code. These tests should be owned by developers and any change in the application code should be supported by component level tests.
Here is a table demonstrating, how a business feature can be tested at component level.
Shared Testing Culture
Teams with an “us vs. them” culture, where QAs handle testing and developers handle code, often struggle with quality in both practices and the product.
Developers, who can write better code, should help QAs in maintaining and making the tests robust. They can educate QAs about various coding practices.
QAs should pair with developers to identify edge cases and impacts early, so developers can consider them while writing code.
A shared testing culture solves many other issues, like reducing the back-and-forth of finding defects, less number of defects from UAT.
