Unit tests

Unit tests are good! They're cheap to write, catch real bugs, but most importantly, force you to write reusable, modular components (as they're the most testable). At Khan, we strongly encourage all code to have associated unit tests.

What is Unit Testing?

Unit testing is a method of testing individual units of source code to verify that they behave as intended. A "unit" is the smallest testable thing. In Python a unit might be a function, a class, or a module. Check it out on Wikipedia.

Unit testing is different from and complements integration testing, systems testing, performance testing, and acceptance testing.

Why Unit Test?

There are many benefits that arise from practicing unit testing.

Tests are Documentation

  • The tests provide an example of how to use the code.
  • Good test names indicate the code's expected behavior.
  • This executable documentation is either up-to-date, or tests are failing.

Make Changes with Confidence

  • Code can be drastically refactored while tests verify that it still works.
  • Tests alert you when the code's behavior is accidentally altered.
  • Found bugs can be encoded as tests. The fix is valid when the tests pass.

Improve the Design of Code

When following the practices of test-driven development (TDD), there are additional design benefits to unit testing. When you write the tests first, you:

  • cause interfaces to be built from a user's perspective.
  • promote thinking through desired behavior and error cases.
  • promote code that can be tested easily.
  • promote code that is well-factored into pieces of reasonable size and complexity.

Best Practices for Unit Tests

  • Test names should be descriptive. When a test fails, you should have a good idea of what went wrong from the name of the failing tests.
  • Tests should be small and focus on one thing. Then, when a test fails, it is evident which code caused the failure.  Large tests are an indication that too much code is being exercised.
  • Tests should be fast. The faster the tests, the less it removes you from your workflow and the more natural the cycle of writing tests, writing code.
  • Tests should be isolated. Don't interact with real databases or network services. Instead use fake objects that you control.
  • Tests should be deterministic and return the same result every time. Don't rely on the clock. Seed random number generators.

Testing Python App Engine Code

Set up your Environment

First, follow the instructions to set up virtualenv. Then make sure the testing libraries are installed.


$ cd webapp   # the root of the stable tree, or your branch of it
$ pip install -r ./requirements.txt

If this fails with errors like:

cc_: _configtest.cc
clang: error: unknown_argument: -mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]

Then you have clang version 5.1 or higher, which treats unknown GCC arguments as errors. To fix, this, run the following, and then run pip again:

$ export ARCHFLAGS="-Wno-error=unused-command-line-argument-hard-error-in-future"

Otherwise, if your pip install fails on numpy, try:

CC=gcc pip install -r requirements.txt

Finally, verify that everything works by running the existing test suite.

$ cd webapp
$ tools/runtests.py

Running Specific Tests

The runtests.py script also allows specifying a directory, a tests file or even a specific test within a file. This is handy for speeding up your test/code cycle by only testing a specific feature or project that you're working on.

$ tools/runtests.py models_test.py
$ tools/runtests.py unisubs/ 
$ tools/runtests.py api.v1.test.user_test.V1TestUser.test_user__students__progressbystudent

Conventions for New Tests

Test files have the form <name>_test.py where <name> is the name of the module being tested, e.g., models.py and models_test.py. This has the nice property that a chunk of code and its tests appear side-by-side when sorted alphabetically.

Within the root webapp directory, you can list all test files using:

$ find . -name '*_test.py'

Within the test file are classes that end in "Test" and inherit from unittest.TestCase. The methods in classes are all prefixed with test_. This enables the top-level test runner to find your tests automatically. For example:

import unittest
class ModelsTest(unittest.TestCase):
  def test_return_video(self):
    # test code


Supporting Libraries

The testutil module contains some wrappers for testbed stubs, like GAEModelTestCase, and other utilities such as MockClock for faking datetime and gaetasktime for working with task queue scheduling times.

The following third-party libraries are available:

  • unittest - Python's standard unit testing framework.
  • Mock - Provides mock objects and object patching.
  • testbed - App Engine comes with testbed which provides service stubs.
  • agar test - Convenience wrapper on top of App Engine's testbed.
  • webtest - Wrapper for WSGI applications.

Testing the Exercise Framework

Read https://github.com/Khan/khan-exercises/wiki/Testing-Exercises

Testing JavaScript

JavaScript tests are written against the Mocha framework with the BDD interface and using chai.js' expect(this).to.eql(that) syntax for assertions.

Adding New Tests
To add a new test, make a new file adjacent to the component being tested ending in _test.js.

For example, tests for javascript/scratchpads-exec-package/output.js should be found in javascript/scratchpads-exec-package/output_test.js,

An Example Test

Let's say you have the following in javascript/shared-package/sum.js:

var _ = require("underscore");

var sum = function() {
    return _.reduce(_.toArray(arguments), function(a, b) {
        return a + b;
    }, 0);
}

module.exports = sum;

The corresponding test would live in javascript/shared-package/sum_test.js:

var sum = require("./sum.js");
var testutil = require("../testutil.js");

var describe = testutil.describe;
var it = testutil.it;
var expect = testutil.expect;

describe("sum", function() {
    it("returns 0 for no arguments", function() {
        expect(sum()).to.eql(0);
    });
    it("returns the argument for 1 argument", function() {
        expect(sum(1)).to.eql(1);
    });
    it("returns the sum of all arguments for many args", function() {
        expect(sum(1, 2, 3)).to.eql(6);
    });
});

If your test depends on a language-specific file or a file that differs between dev and prod, you can use requireWithVars:

var icu = requireWithVars("./third_party/icu.{{lang}}.js");

...

describe(icu, function() { it("formats dates", function() { expect("2014-12-13", icu.formatDate()); }); });

Inline unit tests

A controversial experiment is being run to enable inline unit testing in coach reports. An example function and inline test is below. A vote on whether to continue the experiment will occur on March 31st. See coach-reports_test.jsx for details.

/**
 * Arguments:
 *  - data: A student object returned from /api/v1/user/students/progress
 *  - field: A value of a serialized UserExercise object, such as "key"
 *
 * Returns:
 *  - a list of serialized UserExercise objects, indexed by field
 */
var skillsBy = function(data, field) {
    return _.chain(data)
        .reduce(function(memo, value, key) {
                if (_.contains(Reports.exerciseStates, key)) {
                    if(value.length) {
                        Reports.assert(field in value[0]);
                    }
                    memo = memo.concat(value);
                }
                return memo;
            }, [])
        .indexBy(field)
        .value();
};
/*unittest {
    var input = {
        struggling: [
            {
                key: "k1",
                param2: "2"
            },
            {
                key: "k3",
                param2: "4",
            }
        ],
        mastery1: [
            {
                key: "k5",
                param3: "6"
            }
        ],
        notAMasteryLevel: [
            {
                key: "k7",
                param2: "8"
            }
        ]
    }
    var output = skillsBy(input, "key");
    Reports.assert(_.isEqual(output, {
        "k1": {
            key: "k1",
            param2: "2"
        },
        "k3": {
            key: "k3",
            param2: "4"
        },
        "k5": {
            key: "k5",
            param3: "6"
        }
    }));
}*/

Testing React components

React contains a fantastic testing add-on that lives in React.addons.TestUtils. Of note are TestUtils.renderIntoDocument that creates a component on a detached span (i.e., the name is misleading), and TestUtils.Simulate.{click, ...}.

Careful: Functions that begin with find expect there to be exactly one element found, and will throw an error otherwise. If you expect there to be more than one element found, use functions that start with scry

// Verify that clicking on 'progress' invokes the callback.

var TestUtils = React.addons.TestUtils;

var called = false;
var topic = TestUtils.renderIntoDocument(Topic({
    initiallyExpanded: true,
    name: "Software Engineering",
    skills: [
        {
            translatedName: "Test Making",
            startStatus: "struggling",
            endStatus: "mastery3",
            totalDone: 8,
            timeSpent: 40
        }
    ],
    showProblemsFn: function() {
        called = true;
    }
}));

// Also asserts that there is only one.
var target = TestUtils.findRenderedDOMComponentWithClass(
    topic, "tooltip-target");

assert(target);
TestUtils.Simulate.click(target);
assert(called);

Running Tests

Tests are run using tools/runjstests.py.

You can run all tests within a directory tree, e.g. tools/runjstests.py javascript/shared-package

You can also run all the tests in a specific file, e.g. tools/runjstests.py javascript/shared-package/poppler_test.js

By default, the tests are run headlessly using PhantomJS, but sometimes it's useful to run them in browser for debugging.

You can do this using the --runner or -r flag for short.

tools/runjstests.py --runner browser

Adding Missing Dependencies

When writing new tests, you may find ReferenceErrors to global variables during test runs, despite your module working just fine when used on the website. This is because the test system will only include files in the dependency tree of tests. Dependencies are specified via calls to require(). If a file is not required by the tests or recursively by one of its dependencies, that file will not be loaded in the tests.

For example, in the above sum.js, if you had omitted the var _ = require("underscore");, the code would've worked just fun when run through the site since _ is globalized, but you might've gotten an "_ is not defined" error when running tools/runjstests.py javascript/shared-package/sum_test.js.

If you find yourself trying to test an existing module still relying on global variables instead of require()'ing its dependencies, you'll need to convert that module to use require() before you can write JavaScript unit tests for it.

More details on our require() system can be found in javascript/shared-package/ka-define.js.

Comments