Measuring test effectiveness

The main objectives of testing are to establish confidence and to find defects. This article describes some measures for test effectiveness.

Defect Detection Percentage (OOP)

Whenever a piece of software is written, defects are inserted
during development. The more effective the testing is in finding those
defects, the fewer will escape into operation. For example, if 100 defects
have been built into the software, and our testing only finds 50, then its
DDP is 50%. If we had found 80 defects, we would have a DDP of 80%; if
we had found only 35, our DDP would only have been 35%. Thus it is the
escapes, those defects that escaped detection, which determine the quality
of the detection process. Although we may never know the complete total
of defects inserted, this measure is a very useful one, both for monitoring
the testing process and for predicting future effort.
The basic definition of the Defect Detection Percentage is the number
of defects found by testing, divided by the total known defects. Note that
the total known defects consists of the number of defects found by (this)
testing plus the total number of defects found afterwards. The scope of the
testing represented in this definition may be a test phase such as system
testing or beta testing, testing for a specific functional area, testing of a
given project, or any other testing aspect which it is useful to monitor.
The total known defects found so far is a number that can only increase as
time goes on, so the DDP computed will always go down over time.

DDP at different test stages or application areas DDP can be measured at different stages of software development and for different types of testing.
Unit testing. Because testing at this stage is usually fairly informal, the
best option is for each individual developer to track his or her own
DDP, as a way to improve personal professionalism, as recommended
by Humphreys (1997).
Link, integration, or system testing. The point at which software is
turned over to a more formal process is normally the earliest practical
point to measure DDP.
Different application areas, such as functional areas, major subsystems, or
commercial products. The DDP would not be the same over all groups, as
they may have different test objectives, but over time this can help the test
manager to plan the best value testing within limited constraints.
DDP of early defect detection activities such as early test design and
reviews or inspections. Early test design (the V-model) can find defects when they are much cheaper to fix, as can
reviews and inspections. Knowing the DDP of these early activities as
well as the test execution DDP can help the test manager to find the
most cost effective mix of defect detection activities.