TestNG IRetryAnalyzer is a dependable solution to easily rerun failed tests.
In the modern agile world, where automated testing is a crucial part of delivering software faster, with developers checking in smaller pieces of code more frequently, and continuous integration builds running automated tests, the need for quick and accurate reporting has never been more critical.
While reporting automated test results may sound simple, many teams struggle, specifically in UI automation, dealing with the false positive test result. Particularly at larger organizations, where several product teams share environments, data, and have service interdependencies, false positive test results tend to appear intermittently, and running the exact same test again might result in a passed test. While the test failure may be valid at that moment in time, you usually don’t want to create a bug for something caused by a deploy to an environment, or an unreproducible UI render, and you definitely don’t want to be holding up a release because you can’t determine if the new feature caused the failures or not.
Luckily, many testing frameworks provide built in solutions to solve this common obstacle. One I’ve found to be extremely robust and dependable is testNG’s IRetryAnalyser, which allows you to rerun a failed test method a set amount of times before declaring it as failed. So, without further ado, let’s dive into how we can actually implement TestNG IRetryAnalyzer.
First let’s look at the interface, with only one method, retry will be called if a test method fails, and allows you to get any test details from the ITestResult input argument. This method implementation should return true if you want to re-execute a failed test, and false if you do not want to re-execute. The implementation of this interface determines how many times to retry a failed test based on a fixed counter, which you can see from the below example is the maxRetryCount set at 2 retries.
How to Include IRetryAnalyzer in your tests
Now, let’s see how we can actually use the retryAnalyzer in our tests. If you want to quickly see how it works, you can add the retryAnalyzer value in your @Test annotation like below:
And if we run this test, we will get the following results:
You can see the test ran 3 times, and was not marked as failed until after the two retries. While adding the retryAnalyzer at the individual test level is a decent way to quickly see how the analyzer works, it’s not a great long term solution as it overcrowds your tests annotations, and is just not necessary. That is why, I would suggest implementing the retryAnalyzer at the test suite level, in your @BeforeSuite method. As you can see below, adding to your @BeforeSuite is simple, using TestNG’s ITestContent to apply the analyzer to all test methods via getAllTestMethods.
After applying to your @BeforeSuite, you can remove the retryAnalyzer annotation from your individual tests like below:
And after running again, we will get the same test results:
The TestNG IRetryAnalyzer Difference
This simple, but impactful utilization of testNG’s IRetryAnalyzer is a fantastic way to show increased value from test automation. Implementing the retryAnalyzer, specifically at the test suite level, gives not only your testing team, but entire product team immediate returns.
As a tester, the first noticeable improvement will likely be the added accuracy to your test automation reporting. Long term, and most importantly, you will save your team, and organization money by not wasting time trying to debug inaccurate test results, inevitably accelerating your team’s product lifecycle.
TestNG IRetryAnalyzer & Automation Reporting with tapQA
The test automation consultants at tapQA specialize in powerful, effective QA automation strategies and techniques. If you would like to learn more about TestNG and how IRetryAnalyzer can help your organization, we would love to hear from you.
Aaron Gibbons is a Senior Automation Engineer who has been with tap|QA for over 7 years. He has extensive experience in Quality Assurance and Test Automation. He has implemented quality solutions for many of tap|QA's clients.
Have a QA question?
Our team would love to help!