Test

Unit Testing

Unit testing tests individual units of code. For any new code written, you have to write unit tests for it. Make sure you have written tests to cover the different scenarios handled by the code and cover all the lines of code. We use nose Python package for unit testing. Install the package (if not installed), by following the instructions here. To perform unit testing, follow the workflow below:

1. Change into the 'tests' directory from the project root.

2. For each module, create a file 'tests/test_<module_name>.py' (if it does not exist). This file will contain the unit test cases for that module.

3. Group the test cases for each code unit (e.g., a function). Define a test class for each code unit. The test class will contain the unit tests for that code unit. An introduction to writing unit tests using nose can be found here. A sample test class containing unit tests to test delimiter based tokenizer is shown below:

class DelimiterTestCases(unittest.TestCase):

def test_delimiter_valid(self):

self.assertEqual(delimiter('xyz abc'), ['xyz', 'abc'])

def test_delimiter_none_delim(self):

self.assertEqual(delimiter('xyz abc', None), ['xyz', 'abc'])

@raises(TypeError)

def test_delimiter_inp_none(self):

self.assertEqual(delimiter(None), [])

You can find more sample unit test cases here.

4. Finally, you can run all the unit test cases using the following command executed from project root:

nosetests

Note: you can run unit tests in a specific file using the following command (executed from project root):

nosetests <project-name>.tests.<test file name>

5. Make sure that all the tests pass. If not, revisit the code or the tests to fix the issue.

Regression Testing

Regression testing aims to verify that the code that was previously developed and tested still performs correctly after it was changed or interfaced with a new code. Specifically, regression testing runs all the unit tests that were written for previously existing functionalities. We use nose Python package for regression testing. Install the package (if not installed), by following the instructions here. To perform regression testing:

1. Make sure that there is a directory called 'tests' containing unit test files.

2. Next, run the tests using the following command (in command prompt from project root):

nosetests

3. Make sure all the tests pass. If not, revisit the code or the tests to fix the issue.

Performance Testing

Performance testing aims to determine the responsiveness, throughput, reliability, and/or scalability of the code under a given workload. We use asv Python package for performance testing. To perform performance testing:

1. Create a directory called 'benchmarks' (if it does not exist) under the project root directory. Also, create a file called asv.conf.json (if it does not exist) in the project root directory, containing the configuration information to run the benchmarks. A sample file can be found here.

2. For each module, create a file 'benchmarks/benchmark_<module_name>.py' (if it does not exist). This file will contain the performance tests for that module.

3. Group the test cases for each code unit (e.g., a function). Define a test class for each code unit. The test class will contain the performance tests for that code unit. An introduction to writing performance tests using asv can be found here. A sample test class containing performance tests to test Levenshtein distance measure is shown below:

class TimeLevenshtein:

def time_lev_short_len_strings(self):

levenshtein('short string', 'short string')

def time_lev_med_len_strings(self):

levenshtein('Developing Python Package is fun', 'It is fun to develop a Python package')

You can find more sample performance test cases here.

4. Finally, you can run the performance tests using the following command (in command prompt from project root):

asv run

Note: you can run performance tests specific to a module using the following command (executed in command prompt from project root):

asv run --bench benchmark_<module_name>

5. Make sure that the tests run successfully and the performance (i.e. runtime) is as expected. If not, revisit the code or the tests to fix the issue.

Continuous Integration

Continuous integration tools automate the process of testing a Python package across different Python versions and OS. Specifically, for any change made to a Python package, continuous integration tools would automatically run the unit test cases and report if there are any errors. There are several continuous integration tools for Python. In our group, we use Travis-CI for Linux and MacOS, and Appveyor for Windows. We link these tools to project's GitHub repository (so for any change in the repository, the unit tests are automatically run). The tests are run on the servers maintained by the developers of Travis-CI and Appveyor tools. Both the tools are free for open-source projects.

To set up Travis-CI for your repository follow the workflow below:

1. Create a file called '.travis.yml' in the project root directory. A sample configuration file can be found here.

2. Go to the Travis-CI site and login with your GitHub account to activate testing for your project. Specifically, activate your project in your profile settings. From now on, your project's tests will be run on every push to GitHub.

To set up Appveyor for your repository follow the workflow below:

1. Create a file called 'appveyor.yml' in the project root directory. A sample configuration file can be found here.

2. Go to the Appveyor site and login with your GitHub account to activate testing for your project. Specifically, add your project from the GitHub account for testing. From now on, your project's tests will be run on every push to GitHub.

Resources

• nose python package

• Tutorial for writing unit tests in nose

• Sample unit tests

• asv python package

• Sample asv.conf.json file

• Tutorial for writing performance tests in asv