White Box Testing

We previously looked at Black Box Testing, a method of testing methods, classes or systems that relies solely on the design specification. Black box testing is incredibly useful for finding out which valid or invalid cases your code covers. It can also be designed before your code has been written, which makes it an excellent first step in the design process. On the other side of the spectrum is White Box Testing.

White Box Testing

The main purpose of black box testing is to simulate what happens when a user knows how your system is supposed to work and tries to break it. They are looking for holes or gaps in implementation that produce errors. White box testing assumes that the implementation is correct and focuses on rigorously testing that implementation.

The white box testing is accomplished through code coverage. Code coverage is a general term that is meant to signify the percentage of lines of code covered by your tests. In order to achieve code coverage, however, you MUST know the code in great detail. This is the main way white box testing differs from black-box testing. As a result, white box testing is usually more thorough than black-box testing.

Code Coverage

There are many techniques used to achieve code coverage. Each one is used for different purposes, but only when they are all used together is full 100% coverage possible.

Statement Coverage AKA Line Coverage

Statement coverage is the easiest type of code coverage necessary in white box testing. To achieve statement coverage, every line of code must be executed at least once. In most cases, the set of black-box tests provided by the design team will cover 90-100% of the lines of code. The remaining lines of code would be covered, usually, by tests which cause errors (exceptions in Java). Occasionally, it is impossible to achieve 100% statement coverage. When that happens, just skip it.

Branch Coverage

Branch coverage is a stronger version of statement coverage. It aims to test every branch of code once. This includes:

  • If statements
    • True AND False branches
  • Switch statements
    • every value
    • default case
    • non-value, if no default exists
  • Loops
    • 0 executions and > 0 executions

Example:

Consider the following code (and flowchart):

int a = ... // Some integer input value
int b = ... // Some integer input value
if (a + b > 10) {
    System.out.println("a + b is large");
}

if (a > 5) {
    System.out.println("a is large");
} else {
    System.out.println("a is small");
}

Here we have two if statements. The first is a simple if, but it still has 2 possibilities: a + b > 10, or a + b <=10. That means in total, there are 4 possibilities:

  1. a + b > 10 and a > 5
    • i.e.: a = 6, b= 5
  2. a + b > 10 and a <= 5
    • i.e.: a = 3, b = 9
  3. a + b <= 10 and a > 5
    • i.e.: a = 7, b = 1
  4. a + b <= 10 and a <= 5
    • i.e.: a = 4, b = 4

What we need is 2 cases out of these 4 possibilities that cover all branches (i.e. arrows) on the flowchart. If we choose Test 1 we cover both true branches. With our second test, we would then need to cover both false branches, which is accomplished with Test 4.

Questions:

  1. Why is code coverage not the same as branch coverage?
  2. What makes branch coverage stronger?

Compound Condition Coverage

An even stronger (and far more important) version of branch coverage is compound condition coverage. Let's say you have the following if condition:

if (!done && (colour.equals("blue") || value > 100)) {

It is not enough to simply use 2 test cases; one which skips the if (false) and one which enters (true). Instead, we are going to create 2^n True/False test cases for all simple expressions combinations within our compound condition. In this case, our simple conditions are:

  • !done
  • colour.equals("blue")
  • value > 100

Meaning, we will have 2^3, or 8 cases for this if statement alone. It is now easier to simply create a truth table to ensure we hit all of our possibilities. It is important to note: Sometimes, it will be impossible to satisfy all cases. When that happens ignore the impossible cases. (Question: which of these are "impossible" tests?)

You can see that this type of testing could quickly become unfeasible. It would be your judgement which of the tests can be omitted without compromising the quality of the testing. One approach, called partial compound condition coverage is to choose tests so that each expression is tested as either true or false. In our above case that would be:

  • !done - True and False
  • colour.equals("blue") - True and False
  • value > 100 - True and False
  • colour.equals("blue") || value > 100 - True and False
  • !done && (colour.equals("blue") || value > 100) - True and False

Some of these expressions can be covered with the same test.

Questions: What would be the list of tests to achieve partial compound condition coverage of the following statements:

  • (i < j || k != m)
  • (name != null && (name.equals("Punch") || name.equals("Judy"))
    • Consider what would happen if we changed the above to ((name.equals("Punch") || name.equals("Judy") && name != null)
  • (((u == 0) || (x>5)) && ((y<6) || (z == 0)))

Loop Coverage

Much like if statements, loops have conditions, and as such there are multiple ways to test a loop. There are six major ways of testing loops:

  • Skip the loop (Zero Pass)
  • One through (Single Pass)
  • Twice through (Double Pass)
  • N-1 Pass (may not be possible)
  • N Pass (may not be possible)
  • N+1 Pass (may not be possible)

Where N is the maximum number of times the loop can be executed.

Each of these cases is necessary in order to prove the value and correctness of the loop. Notice that, much like black box testing, we are testing near the limits. This is because that is where most of the errors occur.

Consider the following code. The readInts method takes in a scanner, an array of integers called the buffer, and an integer representing the length of the buffer. If we wanted to test the readInts method to ensure that the loop is executed the correct number of times, we would need the following test cases:

  • Skip loop entirely: bufLen = 0
  • Loop executes exactly once: buf = new int[1 or more], bufLen = 1
  • Loop executes exactly twice: buf = new int[2 or more], bufLen = 2
  • Loop executes n-1 times: buf = new int[n], bufLen = n-1, for some n > 0
  • Loop executes n times: buf = new int[n], bufLen = n, for some n > 0
  • Loop executes n+1 times: buf = new int[n], bufLen = n+1, for some n > 0
  • Loop executes 2 < m < n times: buf = new int[n], bufLen = m, for some m = n / 2 (For arguments sake)

Questions:

Consider the following method:

public static void foo(int i, int n, int x, int[] v) {
    while (i < n && v[i] < x) {
        if (v[i] < 0) {
            v[i] *= -1;
        }
        
        System.out.println(v[i]);
        ++i;
    }
}

In order to test this method we need the ability to ensure that the loop will execute 0, 1, 2 and multiple times. Consider the code presented below.

  1. Ensure that it is tested in the main function with executions 0, 1, 2, and 6 times through the loop.
  2. Is there a possible way to create an n-1, n, and n+1 test for foo? If so, what would be the values of i, n, x and v?