Pairwise testing is a very powerful technique that helps a lot test analysts to speed up the test execution especially when they have to test one system or application with very entries and big test configurations.
The specification-based “Pairwise testing” technique can help a lot test analysts during test execution when test cases has multiple entries and consequently multiple combinations.
Normally, the test planning and execution in projects is ever short and analysts have to choose wisely the “correct” combinations in order to find as much defects and as soon as possible. To address this problem and guarantee the quality the release under testing, test analysts have to resort to specific techniques.
In this case, the most suitable technique would be “Pairwise Testing”. Basically, it consists in combining all possible entries of one specific system, using a smarter possible way in order to generate minimum test outputs as possible and maximizing the variation of the system entries. As this technique reduces the test cases combinations, it reduces the cost of the test execution. Pairwise Testing can be done manually using the “Orthogonal Array” concept or by some specific tools using the “Classification Tree” concept.
This technique has been used along the years ago in other areas as agricultural and medical experiments. In the 20s, some researchers started to use this technique in technology information. Along the years, the usage of this method has been brought reduced costs and quality to the projects.
Orthogonal Array technique:
This technique is done manually using pre-defined arrays in order to generate the test cases variations using the input system variables. Firstly, we have to choose the most suitable array. The number of the columns of this array will be the number of entries plus one. For example, if we have three entries, the selected array will have four columns.
Secondly, we take the entries possibilities and multiply the two largest numbers. The result of this operation will be the number of the lines of the array and meaning the number of possible test cases. After the array is selected, we have to do the correlations between the meaning of each line and column of the array. The number in the column means the order of the entries and the number in the line means the order the variation of the entries. After do the correlations, the test analyst can also remove one test combination if it is invalid. This must be done with very carefully.
Below, I will show an example of this technique. For example, we have to test one application, using the follow environments: Operational Systems (Mac, Linux, Windows 7); Browsers (Firefox, IE and Chrome) and in following Databases (Oracle, SQL Server and MySQL). As we have three entries, the suitable array must have four columns (3+1=4). In relation with lines, the array will have nine ones, because we have three variations of operational systems and databases as maximum entries variation (3*3 =9), meaning that nine test cases will be generated. The selected array is shown below (the value of the arrays is got from international standards).
Table 1: Orthogonal Array
Doing the correlations, the result of combinations is shown below.
Table 2: Test Cases Generated
Analyzing the results above we see that is impossible to test the combinations 2 and 5, because Internet Explorer is not supported with Mac and Linux operational systems. Only after that, quality analysts can remove these configurations from the test matrix above. Then, the final test combinations got from the technique is seven different combinations.
If test analysts do not use this technique, a lot of test combinations had to be done in order to guarantee the minimum quality of the product under testing. For this test, quality analysts will have to test twenty seven different environment combinations, because the supported environments are three versions of operational systems, browsers and databases (3*3*3=27). This number was reduced to only five guaranteeing the needed quality associated to this technique. Translating to the cost of the quality more than eighty per cent was reduced using this technique!!!
Classification Tree technique:
Instead raising the test combinations using orthogonal array manually, test analysts can use some tools that implement other technique that closely resemble the orthogonal array, the classification tree technique. This method is also used when test analysts have to test one system or feature with a lot of possible entries and have to test all combinations to find as many defects as soon as possible. To test all combinations would demand a lot of time and normally the time to test applications is very short.
Classification Tree technique has as base technical the theory that when pairs or triples of the variables of the system are combined and tested together, high percentage of the defects is quickly found (around 95% of all defects).
This theory is also used by other specification-based technique named “Equivalent Partitioning” and “Boundary Value Analysis”. Using these methods some results can be grouped into equivalent classes. Those classes have the same output characteristics when using a range of defined values. For example, to be approved one student has to get a score greater of equal to sixty. All test values between sixty and one hundred will exercise the equivalent class “approved” and values from zero to fifty-nine will exercise the class “reproved”. The theory associated with those is that where the most of the defects are found is exactly in the boundary values and to guarantee the coverage of the class, two or three values must be chosen. This quantity of the values is exactly the numbers that the Classification Tree technique implements, pairs or triples of input combinations.
There are a lot of available tools that implements this technique and test analysts can easily see graphically the optimum test scenarios when combining all pairs or triples of the entries of the system. I chose the best known named “CTE XL” tool (I will mention in the end of this article in References part how to get this tool).
To illustrate this technique, I will create a test example in which test analyst has to test one interface responsible to record user information. This interface has some numeric, alpha and date fields as Name, Phone, Address, City, Cep as pictures below show. First picture illustrate the test cases generation using pairs of entries and the second one illustrates test cases using triples of entries.
If triple entry combination will be used, more test cases will be generated. Test analysts have to find the balance between how much time is available to complete the testing and the risk associated in choosing one technique in comparison with other one.
Figure 1: Test Cases Generation using pair of entries
Figure 2: Test Cases Generation using triple of entries
Following are some references which can be followed to get more idea about pair wise testing.
- Advanced Software Testing - Vol. 1, Rex Black.
- Presentations on Star East conference:
- Pairwise Testing Comes of Age, George B. Sheerwood, Ph.D.
- The Bug Stops Here (Improving Test Design with a Novel Test Case Generator), Technical Presentation, Salon 6, May18.
- Download the tool.