The following is some information describing the benefits of Beta Testing.
There is a lot of data available to support these following claims —
System testing (including both the internal black-box feature testing after
the developers have done their own unit testing, and the external beta
testing), consumes about 25 – 33% of the total system development budget.
Of this, beta testing typically consumes 10% to 25% of the system testing
budget (i.e., 3% to 8% of the entire system development project budget).
The number of defects delivered to system testing is about per 25 – 35
defects per 1,000 lines of source code (LOC), in situations where beta testing
will be employed.
The number of defects remaining after fairly extensive internal system
testing is about 5 to 10 defects per LOC, prior to the beta.
The remaining defects after an effective, fairly extensive beta is about 1
to 3 defects per 1,000 LOC.
The cost per defect is about $15,000 to $30,000, including higher support
costs, lost sales, litigation, etc. Not all these costs may be borne
directly by the software supplier; some may be “swallowed’ by the clients in
terms of lost productivity. Eventually, of course, these costs will all be
incurred directly or indirectly on the software supplier as the birds come
home to roost.
It costs approx. $100,000 to $150,000 in the U.S. to deliver 1,000 LOC in C
or Java., including all costs for requirements definition, design, coding,
testing, debugging, documentation, etc.
So based on these numbers, we can conclude —
(1) Extensive beta testing costs about $4,000 to $8,000 per 1,000 LOC.
(2) Beta testing uncovers about an additional 3 to 6 bugs per 1,000 LOC.
(3) The costs avoided by finding these bugs is about $50,000 to $75,000.
So therefore the payback for every $1 spent on beta testing is approximately
$8 to $12.
Please note that conservative numbers have been used throughout this analysis. In
other words, these claimed savings are probably at the bottom of the range
of what is likely to be experienced.