All-Pairs Testing – Bridging the Gap Between Minimal Test Cases and Software Quality

Posted on Mar 13 2010 - 5:48am by Raj

The only universal statement that will satisfy the diverse software development principles of IT professionals is that the software to be delivered to clients must have the highest accuracy and reliability as possible. Software testing was proven to be a vital part of achieving such results since the early days of IT industry up to present situation when numerous testing methodologies have already been developed. One of the most widely adopted software testing techniques is all-pairs testing.

In this method, input parameters will be combined with one another to create all discrete possibilities that may occur during operation. Contrary to exhaustive testing, or verifying all combinations individually, this method utilizes statistical principles to find software faults and to reduce a number of test cases while maintaining satisfactory test coverage. In actual practice, this often translates to higher productivity and lower cost.

Real-life software testing always faces the dilemma of having thousands of test cases to thoroughly verify internal and external system behavior. However, meeting client needs often restricts project management in having the luxury of time during testing phase to cover all test cases. Thus, different types of all-pairs testing was formed to incorporate both past and modern statistical concepts into test case creation to find as many faults as possible out of minimal cases.

Combinatorial Testing
This testing is the simplest pair testing. In this type, all combinations should be evaluated by tedious approaches like prime-based solutions and brute-force. Such solutions where proven to be tiresome and least reliable because it relies on random probabilities and guesses from previous results. Apart from this, it imposes higher chance of error because of its heavy testing procedures and its similarity with exhaustive testing.

Pairwise Testing
In exhaustive testing, all combinations are being formulated. For example, 243 items should be formulated for a system with five distinct parameters having three possible states. Testers do not have enough resources to cover all 243 items but through the pair wise testing, only minimal test cases will be needed to cover all interactions. Finite field computation, modulo arithmetic, etc. is used to solve the least number of test cases needed. This testing started as early as 1990s and continues to become popular because of its feasibility and high quality despite its low cost.

Orthogonal Array Testing
This type was derived from industrial engineering concepts of manufacturing. It offers robust methodologies in creating efficient test cases. This pair testing is similar with the pair wise but goes further by arranging each combination orthogonal. In the orthogonal arrangement, combinations of values between parameter pairs will only occur once in all sides of the array. This arrangement makes it suitable for more sophisticated software applications because of faster and more reliable bug detection.

Through pair testing is appropriate for most systems, it will not be as efficient when applied in medical equipment, distributed computing, and shared databases. All-pairs testing common mistakes can be summarized into three:

Manual Pair Creation
Using spreadsheets to manually plot pair values is very tedious. It is also difficult to maintain and presents lots of inaccuracies. To avoid such scenarios, test engineers can opt to professional or freeware applications to automate their test case creation.

Wrong Application Usage
This is either using pair testing in wrong application, or using it in correct application but in its wrong area. Pair testing is not applicable to high-risk applications like medical systems. It should be used in collaboration with other testing procedures to ensure its reliability. This is because the high-risk systems encounter fatal errors from useless combinations being ignored by pair testing. Wrong usage also includes employing pair testing in areas where simpler methods with less effort are more applicable.

Wrong Parameter Selection
Success of pair testing depends on how well the parameter interaction is defined. Using wrong parameters in pairing will yield low success rate. Sometimes, choosing the best test parameters calls for involvement of requirements analyst, developer, and test engineers.

About the Author

Leave A Response