Another step toward test life cycle automation
Associate Director – Intelligent Automation
Effective testing is an essential phase in the software development process. To improve it, companies have begun to shift to extreme test automation. The integration/continuous delivery (CI/CD) pipeline enables higher automation coverage, maximizing the efficiency of the entire process. However, challenges persist. The Software Development Engineer in Test (SDET) often needs help with sifting through automation reports and logs and with debugging the test results. Given advancements in technology, engineers can now get help via a new automated defect triaging solution. This solution/approach is part of a concept known as test life cycle automation, which substantially reduces human error. In addition, it helps enterprises deliver bug-free applications to their users.
This article is the final piece in a three-part series about this topic. The first article, “Self-healing test automation – a step toward test lifecycle automation,” was published in July 2021, and the second, “Automated impact-based test selection and prioritization,” was published in March 2022.
Even when tests become automated, ancillary activities (like script maintenance, impact analysis, and defect triaging) can be left behind. The persisting manual nature of these activities can slow down software development. A need persists for automation that is carried out throughout the testing life cycle and that speeds up the software development life cycle (SDLC). Fortunately, the rise of artificial intelligence/machine learning (AI/ML) techniques and programming capabilities has created a paradigm shift to encompass test lifecycle automation and not just stick to test automation. Now, automation engineers can focus solely on creating new test scripts.
Within the world of test life cycle automation, automated defect triaging (ADT) is attracting interest, and clients are taking notice. Most automation engineers and SDETs follow a three-part process:
This approach is reasonable when test volumetrics are low or when test execution is infrequent. However, it is easy to imagine a scenario in which many automated test scripts need to be executed daily. The automation engineers may become overwhelmed and need an automated mechanism that can aid in defect triaging and logging.
Figure 1: Workflow of an ADT
The ADT solution uses an AI/ML model to auto-categorize failed test scripts into predefined categories based on test execution and failure patterns. It helps identify failed execution root causes and suggests recommended actions based on knowledge from previous executions.
It consists of the following steps:
Step 1: Test script execution result collation
Irrespective of the test automation framework used — whether based on open-source tools like selenium/extent reporter or on commercial off-the-shelf (COTS) tools like UFT — the test results need to be gathered in a commonly accessible location for the ADT agent.
Step 2: Monitoring through ADT agent
The agent will continuously poll the reports and test execution result stack trace to gather all the results and move them to a database.
Step 3: Automated Defect Triaging (ADT)
The ADT solution picks up the data that the ADT agent sends to a database. After initial data preparation, the data is fed into a classifier model, which helps determine the reason for failure. The test failure is classified based on the result description and by pattern-matching with the historical data. This classification determines if the failure was due to a configuration failure, test infrastructure failure, or a product failure. If at any moment the classifier model cannot auto-classify the defect due to low confidence, the human in the loop (HITL) comes into the picture to assist with the workflow.
Step 4: Defect de-duplication
Once the cause of failure is determined, it passes on to the defect de-duplication, and the model tries to identify similar predicate defects that already exist within the defect management tool. If the classifier model cannot succeed because of low confidence, the human in the loop comes into the picture to assist with the workflow.
Step 5: Defect logging
Defect logging is the final step in the process, closing the loop. The solution should update the existing defect if a duplicate is found, or it should log a new defect within the defect management tool.
The output of the ADT solution is extensive, as it improves the following processes:
The right-fit, popular-classification algorithms can be used for various classifier models. Moreover, a proof of concept (PoC) should be performed on existing data in order to determine the right algorithm. The ADT team should work with the quality assurance (QA) team to retrain the model regularly and triage the defects. The ADT solution helps reduce the test script failure triage efforts by a considerable margin. In a sprint model with a high level of DevOps automation, there is constant pressure on the test engineers to perform continuous testing within the sprint, and to do so effectively. Bottlenecks within the manual defect triaging process consume SMEs’ bandwidth — often when they should be focused on testing. This effect is magnified when there are thousands of automated scripts. It would be wise for companies to invest in a solution that addresses these challenges.
Ajit Rajshekar is a test architect with over 15 years of quality engineering experience, mainly in the product engineering domain (medical devices). His interests include exploring and using new methodologies/tools in test automation and cognitive intelligence (AI/ML) in testing.
Subscribe to keep up-to-date with recent industry developments including industry insights and innovative solution capabilities
Uncover the data in your documents with intelligent document processing.