Embedded Files
[Nov 22, 2021]
Not having a clear idea of the black box and the principles around it is often the core reason of many misunderstandings and anti-patterns in unit and integration testing. When you test, there is always something you are testing, this “something” is called the black box. It is also referred to as “subject under test”, “unit under test” etc. I like to call it the black box under test for reasons I will outline below. In any type of testing, defining the black box under test, and sticking to it, is a key to make the tests effective. You might think “but oh what about white box testing?”. Unit and integration testing are not whitebox tests. Whitebox tests are a way to expose issues in the tests. For example, mutation testing is a form of white box testing that will attempt to introduce bugs in the code, and will warn you if none of the unit tests fail. This means you need to improve your black box tests to catch those issues. If your unit and integration tests are good, whitebox tests should find nothing.
For unit tests, the black box under test is the class. For integration tests, the black box under test will be your service, or set of APIs. For a given system, the first step is to identify what are the black boxes which you will be testing and why. Once you have that, the principles of good testing apply to any black box, hence any type of automated testing.
What is a black box and the parts around it?
Black Box Under Test – This is the subject under test. For unit testing this is a single class.
Inputs - These are the input values and scenarios that can affect the behavior of the black box. If the black box only has one stateless public method, then it will be the inputs of that method. If the black box has multiple public methods, then the inputs will also consist of sequence and ordering of calling the operations. For example, if your black box under test is a Stack class, then calling push followed by pop is a type of input, calling push, push, pop is another type of input etc.
Outputs – These are outputs produced by your black box. You will typically assert on these outputs to see that the correct result is being produced. For stateless operations, it is simply looking at the output of the invoked operation, but for stateful operations, just asserting on the output is not enough. For example, in order to assert that (push push push pop) behaved correctly, the test not only needs to assert on the value of the third pop, but the test also needs to pop and assert twice more to ensure the first two pushes worked. Finally, assert that the stack is empty, and that a subsequent pop fails.
Dependency interactions (Read) – Read dependency interactions are dependencies which your black box interacts with, that affect the output or state of the black box. For example if you have a class that talks to an AuthProvider, and the result of your class varies based on the output of AuthProvider, then AuthProvider is a read dependency interaction. When creating test scenarios, setting up read interaction is a form of input.
Dependency interactions (Write) – Write dependency interactions are dependencies which your black box must interacts with in order for its function to be considered successful, but interaction with that dependency does not affect the output of the black box. For example if you have a class that needs to return the value from the database, and emit a notification to a NotificationProvider class, then NotificationProvider is a write dependency, since the result of NotificationProvider does not affect the output of the black box, but invoking NotificationProvider is a necessary part of the black box’s responsibilities. When creating test scenarios, verifying write interactions is a form of “output”.
Test – This tests the black box based on its behavioral specifications. A behavioral specification is a list of input-output scenarios to specify the behavior of the black box. The tests:
set up the dependency read interactions
provide inputs to the black box under test
assert on outputs of the black box under test
verify the dependency write interactions
Note: There is no need to verify dependency read interactions, since those will manifest in the output of the black box. Secondly, the way to determine read or write interactions is the way they affect the black box, and not based on whether or not they return an output. Dependencies with void return type will almost always be write interactions, but dependencies with non-void outputs aren’t necessarily read type interactions. For example, if the NotificationProvider returns a status code, and the black box is supposed to ignore it and not change its output based on the status code, then NotificationProvider is a write dependency, because all we care about is that it was invoked.
Page updated
Google Sites
Report abuse