Key metrics to measure quality of a Testing Center of Excellence (TCoE)

Shyam Ramanathan,

Global Delivery Director 

Published: May 6, 2015

Establishing a Testing Center of Excellence is a key undertaking for many organizations now. The current spend on quality has increased considerably and it is growing at a faster rate than overall IT spend. A lot of customers want to centralize testing to get a unified, wholesome picture of quality within an organization.

The audit requirements have also become stringent and Quality Assurance (QA) has to be separated from the development. Providing an independent view of quality is of paramount importance, as it has been well said, what gets measured gets improved. It is important not to bombard customers with more measurements, but to identify the key metrics that can help in making appropriate course corrections.

The 'Shift Left' philosophy is the key in enabling a better overall Cost of Quality. This means QA teams also need to be involved right at the beginning of the Software development life cycle to get the maximum benefit. Encouraging QA teams to participate in User Story or Requirement Analysis sessions ultimately ensures better understanding of the requirements which results in better scenarios/test cases.

Following are the key metrics that can help drive QA improvement

  1. Defect removal efficiency (DRE): It is measured by the percentage of all defects detected in the testing phase, over all defects detected, including downstream phases of UAT and production. The key is to have a high DRE. A low DRE represents weakness in the testing process during the earlier phases of the SDLC. We can improve this by doing a root cause analysis of all the defects found in subsequent phases and learning where the gap is. In most cases, lack of business knowledge is at the root of the problem, so increasing training and domain expertise is vital to improving DRE. It has been well established that finding a defect early in the life cycle reduces the overall cost by a huge number.
  2. Test Scripts on time delivery: This metric gives us an idea if the estimation model we used is correct. If we don't meet the schedule it is most likely due to lack of initial planning. This metric gives us an indicator on how to plan for future projects better. It is also important to understand if the earlier deliverables which are inputs to the testing phase were delivered on time.
  3. Test coverage: This metric shows the percentage of requirements covered by testing, over total requirements. This is a critical metric as it tells us which requirements have not got an associated test case. The aim of a QA team is to hit all requirements and ensure maximum coverage. Lack of adequate test coverage can result in defects being found in the downstream phases like UAT.
  4. Test execution metrics: Percentage of Test Case Execution = (Number of Passed Tests + Number of Failed Tests + Number of Blocked Tests) / Number of Test Cases. The percentage of test case execution should be 100%. If it is not equal to 100%, the team must review each and every unexecuted test case to understand and address the gap. This also allows the QA manager to do an analysis and perform Risk Based Testing based on the need of the project.
  5. Defects by severity: This metric provides an overview on how many defects were found across the severity levels defined by the QA program. If more fall under the HIGH category, it is best for the development team to do a root cause analysis and identify actions that can be taken to reduce this occurrence.

Addressing these QA metrics is crucial for creating a well-established Testing Center of Excellence. Reporting these metrics on a regular basis ensures project quality is always maintained at a high level. This is vital to helping customer make project decisions and appropriate course corrections, and will eventually lead to better business outcomes.

Shyam Ramanathan

Global Delivery Director 

Shyam Ramanathan has been working in the IT world for over 21 years, where he has developed a breadth of knowledge and experience in program management, requirements analysis, requirements review, Use case modelling, project planning, test design, test execution, and establishing global delivery teams across geographies and building Center of Excellence. 

Transformative digital technology solutions

Dramatically increase the success of your digital transformation

Related content