RegressionTestSelection

AProject page/notes for regression test selection investigation.

The Regression Test Selection Problem

Statement of Problem

Classical statement of regression test selection problem

Existing Work

Proposal

Still ill-defined! C'est la vie.

Hypothesis/Hypotheses

Existing research doesn't take into account the overhead of the test selection tool
- Maintenance overhead (e.g. amount and fragility of test and source history). A test selection tool should be operationally as simple as possible. Fewer moving parts makes it easier for the tool to prevent mishaps.
- Performance overhead (e.g. how much time does it take to determine whether a test should be executed). A test selection tool should do as little work as possible to determine, safely, whether a test needs to be executed. (One of the papers co-authored by Harrold has some performance analysis in it, but I believe it's a decade years old.)

Motivation

As a developer, I want to run all appropriate unit tests in my product's test suite before delivering to the repository, so I don't break the build and upset my colleagues. However, our unit test suite has become so large that the select all tests technique is intolerable.

This is the motivation behind Kent Beck's JUnit Max, but at first glance its approach seems to be to run tests continuously using different heuristics to determine which tests to run. Also, an IDE plugin doesn't address the important use case of the continuous integration build.

Tools

Regression Test Selection Tools

RETEST out of GA Tech (get a copy of source or binaries for expirical work)

Dependency Analysis Tools

Jikes: Jikes performs a dependency analysis on your code that provides two very useful features: Incremental builds and makefile generation.
Extended Java Compiler (PDF): Report [Lab04b] studies the problem of extracting dependencyinformation from Java sources and proposes and encoding of these dependency information as regular expressions. This encoding is both compact to store and fast to check.
JavaMake: Our make tool keeps track of dependencies between classes and recompiles the source files as necessary by automatically creating and maintaining the so-called project database (equivalent of a "dependency cache" that some other tools use). It contains information about dependencies between classes, and is created the first time you invoke the tool on a given set of .java sources. After that, whenever you change a source file and invoke the make tool, the timestamps of the sources are compared against those preserved in the database, and the changed .java files are recompiled. Then the updated binary classes are checked against their old versions, and, depending on the changes made to them, some dependent classes may be recompiled. If no compilation errors happen, the database is updated, i.e. new class versions get into it, reflecting the new project state.
Nmake: The :JAVA: operator uses dependencies among .java and .class files to insure that build steps run in the proper order, and that only necessary build steps get run. These dependencies are automatically derived from Java source files using the javadeps package. Complex requirements of Java builds, such as dependency cycles and implicit targets, are correctly handled.

Miscellaneous

Unit Test Maintenance Problem

Not only is the time to execute a unit test suite a problem for commercial developers. Maintaining the unit test suite is a problem, too. There are a few issues here. The some Java unit test frameworks represent expected-to-be-invoked methods as strings. Strings are (shallow) refactoring-adverse, so when a method m is renamed to n, and some test T expects "m" to be invoked during its execution, T breaks, even though the entire code base still compiles after m has been renamed. Also, moving beyond renaming methods, if an IDE knows which test classes actually exercise a given class, what changes if any can the IDE automatically make to the test class in response to a change in the tested class?

Testing and Software Development Issues in Internet-Scale Systems

According to James Hamilton (PDF), Amazon's data center guy:

[F]irm separation of development, test, and operations isn’t the most effective approach in the services world. The trend we’ve seen when looking across many services is that low-cost administration correlates highly with how closely the development, test, and operations teams work together.

So the operational efficiency of Internet-scale systems depends to some extent on breaking down the conventional separation of development, test, and operations. Does breaking those barriers suggest any potentially fruitful changes to the way software is developed or tested? Could regression test selection techniques speed up the rate at which web properties deploy? Could version control data be reliably exploited in order to speed up the diagnosis of operational problems? In most cases a web property wouldn't want to instrument every instance of a component that's deployed in a data center, but selective instrumentation (e.g. deploy an instrumented copy of a component to a small fraction of the machines in a data center) could be used to close the loop between operations and development, with runtime (i.e. operational) information being fed back to the development team. (More information on architectural and operational aspects of Internet-scale services can be found in this minibook from two Google data center managers.)

Page updated

Google Sites

Report abuse