CatWalk

CatWalk is a tool for lazy systematic unit testing. It performs bounded exhaustive state and equivalence partition testing of any compiled Java class, supplied by the programmer. It tests for full conformance to a lazy specification, which is learned on-the-fly from the code and hints given by the programmer. The tests learned internally may be exported as a JUnit test suite.

Lazy systematic testing is based on the two notions of lazy specification, in which a specification is inferred at the last minute from frequently modified code, and systematic testing, in which the class is rigorously tested, exploring its whole input and state space, using powerful testing algorithms.

CatWalk testing a BankAccount

CatWalk can be used right from the outset, as soon as a compiled Java class is ready for testing. After loading the class, you can inspect its public constructor and method protocols, or analyse its algebraic structure, and then explore test sequences of increasing path length. You may then validate the test sequences, in which the tool builds a test oracle from a combination of hints given by the tester and inference rules that predict the class's regular behaviour. Afterwards, testing is fully automatic, up to the learned specification. You may also generate an external JUnit test suite, although CatWalk prefers to regenerate tests internally.

You may start by exploring the algebraic structure of the test class, finding which methods force it into new states, and observing how many input partitions the methods have. This way, you may discover the Complete Behavioural Response of the class. You may then generate tests for this, or for all mutating sequences, or exhaustively for all sequences.

Over a million test sequences may be executed, before CatWalk runs out of memory. If the test class is later modified, CatWalk will quickly re-learn the parts of the specifiction that were altered, making it ideal for agile methods, where the code base is expected to change frequently.

There are other Java testing tools. JUnit is familiar to the agile community. Randoop is used by the random-testing community. EvoSuite is well known in the search-based testing community. CatWalk is different from each of these.

• JUnit manual tests are time-consuming to write: you might not cover all the test cases. CatWalk proposes all the right test cases automatically.

• Randoop generates lots of random tests: but you never know what has been covered. CatWalk generates systematic tests, so you know what has been covered, up to what depth.

• EvoSuite generates automatic oracles for its tests: but you cannot tell whether they confirm correct, or faulty behaviour! CatWalk collaborates with the tester to confirm selected test sequences.

• JUnit only automates test execution; and saved regression tests rapidly go out of date. CatWalk regenerates tests when the test class has changed.

CatWalk thinks up the right test cases that even the expert fails to discover. When the test class is modified or extended by subclassing, CatWalk generates exactly the test cases for all novel interleaved combinations of local and inherited methods. Eventually, a fully trained oracle will detect any mutation made to the test class.