Unit tests

Unit tests are good! They're cheap to write, catch real bugs, but most importantly, force you to write reusable, modular components (as they're the most testable). At Khan, we strongly encourage all code to have associated unit tests.

What is Unit Testing?

Unit testing is a method of testing individual units of source code to verify that they behave as intended. A "unit" is the smallest testable thing. In Python a unit might be a function, a class, or a module. Check it out on Wikipedia.

Unit testing is different from and complements integration testing, systems testing, performance testing, and acceptance testing.

Why Unit Test?

There are many benefits that arise from practicing unit testing.

Tests are Documentation

  • The tests provide an example of how to use the code.
  • Good test names indicate the code's expected behavior.
  • This executable documentation is either up-to-date, or tests are failing.

Make Changes with Confidence

  • Code can be drastically refactored while tests verify that it still works.
  • Tests alert you when the code's behavior is accidentally altered.
  • Found bugs can be encoded as tests. The fix is valid when the tests pass.

Improve the Design of Code

When following the practices of test-driven development (TDD), there are additional design benefits to unit testing. When you write the tests first, you:

  • cause interfaces to be built from a user's perspective.
  • promote thinking through desired behavior and error cases.
  • promote code that can be tested easily.
  • promote code that is well-factored into pieces of reasonable size and complexity.

Best Practices for Unit Tests

  • Test names should be descriptive. When a test fails, you should have a good idea of what went wrong from the name of the failing tests.
  • Tests should be small and focus on one thing. Then, when a test fails, it is evident which code caused the failure.  Large tests are an indication that too much code is being exercised.
  • Tests should be fast. The faster the tests, the less it removes you from your workflow and the more natural the cycle of writing tests, writing code.
  • Tests should be isolated. Don't interact with real databases or network services. Instead use fake objects that you control.
  • Tests should be deterministic and return the same result every time. Don't rely on the clock. Seed random number generators.

Testing Python App Engine Code

Set up your Environment

First, follow the instructions to set up virtualenv. Then make sure the testing libraries are installed.

$ cd webapp   # the root of the stable tree, or your branch of it
$ pip install -r ./requirements.txt

If this fails with errors like:

cc_: _configtest.cc
clang: error: unknown_argument: -mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]

Then you have clang version 5.1 or higher, which treats unknown GCC arguments as errors. To fix, this, run the following, and then run pip again:

$ export ARCHFLAGS="-Wno-error=unused-command-line-argument-hard-error-in-future"

Otherwise, if your pip install fails on numpy, try:

CC=gcc pip install -r requirements.txt

Finally, verify that everything works by running the existing test suite.

$ cd webapp
$ tools/runtests.py

Running Specific Tests

The runtests.py script also allows specifying a directory, a tests file or even a specific test within a file. This is handy for speeding up your test/code cycle by only testing a specific feature or project that you're working on.

$ tools/runtests.py bigbingo
$ tools/runtests.py bigbingo/summarize_test.py 

$ tools/runtests.py api.v1.test.user_test.V1TestUser.test_user__students__progressbystudent

If you want to use the pdb Python debugger while running a test, run with the option -j1 so as to run in only one thread. 

$ tools/runtests.py -j1 intl

Conventions for New Tests

Test files have the form <name>_test.py where <name> is the name of the module being tested, e.g., models.py and models_test.py. This has the nice property that a chunk of code and its tests appear side-by-side when sorted alphabetically.

Within the root webapp directory, you can list all test files using:

$ find . -name '*_test.py'

Within the test file are classes that end in "Test" and inherit from unittest.TestCase. The methods in classes are all prefixed with test_. This enables the top-level test runner to find your tests automatically. For example:

import unittest
class ModelsTest(unittest.TestCase):
  def test_return_video(self):
    # test code

Supporting Libraries

The testutil module contains some wrappers for testbed stubs, like GAEModelTestCase, and other utilities such as MockClock for faking datetime and gaetasktime for working with task queue scheduling times.

The following third-party libraries are available:

  • unittest - Python's standard unit testing framework.
  • Mock - Provides mock objects and object patching.
  • testbed - App Engine comes with testbed which provides service stubs.
  • webtest - Wrapper for WSGI applications.

Testing RequestHandler Endpoints

See https://sites.google.com/a/khanacademy.org/forge/for-developers/testing-at-khan/unit-tests/unit-tests-for-webapp2-endpoints

Testing the Exercise Framework

Read https://github.com/Khan/khan-exercises/wiki/Testing-Exercises

Testing JavaScript

JavaScript tests are written against the Mocha framework with the BDD interface and using chai.js' assert.equal(actual, expected) syntax for assertions. For mocking, use Sinon.JS, which is accessible through testutil.sinon.

Adding New Tests
To add a new test, make a new file adjacent to the component being tested ending in _test.js.

For example, tests for javascript/scratchpads-exec-package/output.js should be found in javascript/scratchpads-exec-package/output_test.js,

An Example Test

Let's say you have the following in javascript/shared-package/sum.js:

var _ = require("underscore");

var sum = function() {
    return _.reduce(_.toArray(arguments), function(a, b) {
        return a + b;
    }, 0);

module.exports = sum;

The corresponding test would live in javascript/shared-package/sum_test.js:

var sum = require("./sum.js");
var testutil = require("../testutil.js");

var describe = testutil.describe;
var it = testutil.it;
var assert = testutil.assert;

describe("sum", function() {
    it("returns 0 for no arguments", function() {
        assert.equal(sum(), 0);
    it("returns the argument for 1 argument", function() {
        assert.equal(sum(1), 1);
    it("returns the sum of all arguments for many args", function() {
        assert.equal(sum(1, 2, 3), 6);

If your test depends on a language-specific file or a file that differs between dev and prod, you can use require.withVars:

var icu = require.withVars("./third_party/icu.{{lang}}.js");


describe(icu, function() { it("formats dates", function() { assert.equal("2014-12-13", icu.formatDate()); }); })

Example Tests

To find all the tests in the system if you're looking for some reference, you can search like this:

find . -name '*_test.js'

For more specific examples, here are some suggestions:

Testing React components

The easiest way of testing React components is using fixtures files, described here: React Fixtures and the Sandbox. If you want something more precise, then read on!

React contains a fantastic testing add-on that lives in React.addons.TestUtils. Of note are TestUtils.renderIntoDocument that creates a component on a detached span (i.e., the name is misleading), and TestUtils.Simulate.{click, ...}.

Careful: Functions that begin with find expect there to be exactly one element found, and will throw an error otherwise. If you expect there to be more than one element found, use functions that start with scry

// Verify that clicking on 'progress' invokes the callback.

var ReactTestUtils = React.addons.TestUtils;

var called = false;
var topic = ReactTestUtils.renderIntoDocument(Topic({
    initiallyExpanded: true,
    name: "Software Engineering",
    skills: [
            translatedName: "Test Making",
            startStatus: "struggling",
            endStatus: "mastery3",
            totalDone: 8,
            timeSpent: 40
    showProblemsFn: function() {
        called = true;

// Also asserts that there is only one.
var target = ReactTestUtils.findRenderedDOMComponentWithClass(
    topic, "tooltip-target");


Running Tests

Tests are run using tools/runjstests.py.

You can run all tests within a directory tree, e.g. tools/runjstests.py javascript/shared-package

You can also run all the tests in a specific file, e.g. tools/runjstests.py javascript/shared-package/poppler_test.js

By default, the tests are run headlessly using PhantomJS, but sometimes it's useful to run them in browser for debugging.

You can do this using the --runner or -r flag for short.

tools/runjstests.py --runner browser

To debug tests use the browser developer tools. You can set breakpoints in your code with the debugger statement or you can set break points using the developer tools.