Practical Testing

There is a comprehensive discussion on testing with Python here on realpython (great site!). All of the following topics are elements of practical testing techniques which we use or are used on this course.

Automated Testing

The simplest way to do automated testing in Python is to use the doctest module. The doctest module lets you write your test cases in the docstring of your function or method. This is by far the easiest way to test functions. The only drawback is that it can make docstrings quite long.

# importing the module

import doctest

# function

def numbers_sum(num1, num2, num3):

"""

(num, num, num) -> num

This function returns the sum of all the arguments

>>> numbers_sum(1, 2, 3)

6

>>> numbers_sum(6, 7.0, 8.0)

21.0

"""

return sum(num1, num2, num3)

# invoking the testmod function if this file is being executed

if __name__ == "__main__":

doctest.testmod(name='numbers_sum', verbose=True)

# note that if we had more than one function in the file, we

# can specify just a single function to test, or omit this if

# we wanted to test all functions in that file

# in that case it would just be

# doctest.testmod(verbose=True)

# and leaving out the verbose=True would make it run silently

# doctest.testmod()

# so the only output would be fail conditions.

If you run the above code, you will get the following results.

Trying:

numbers_sum(1, 2, 3, 4, 5)

Expecting:

15

ok

Trying:

numbers_sum(6, 7, 8)

Expecting:

21

ok

1 items had no tests:

numbers_sum

1 items passed all tests:

2 tests in numbers_sum.numbers_sum

2 tests in 2 items.

2 passed and 0 failed.

Test passed.

TestResults(failed=0, attempted=2)

When you run the code it will show if each of the tests pass, or fail. If you do not put in the verbose=True, you will see nothing if all tests pass- you might not even know it tried to test anything.

doctest.testmod() is a simple enough way to test and not very "smart". For example, if you have extra spaces in your sample return/output it will fail when the value created does not match.

Whether using doctest, OR doing manual testing, it is important to consider base and edge cases. You might assume that the user of the function will use args that you assume they will use.

You may have written a game that includes some betting. You run some tests and it all looks like it works ok.

• What if the user enters bets of 5, 10, 50 or 100 Euro? Well, you likely tested these thinking anyone playing would enter something like this. Heck- you even tested with 0.5 for a cheap bet of 50 cents! These are base cases!

• What if the user enters a bet of zero or 1000000000000000000 Euro? These could be edge cases! At the boundary of what is accepted.

• What if the user enters a negative number? Well- that violates a precondition- something you might have assumed would not happen, but a smart user might try it hoping that they lose. Does your code check that this precondition is met? This is a good case to use an assert statement.

• What if the user enters "My dog" as a bet? This cannot easily convert to an amount- this is a TypeError and you have to make sure that your function will either:

  • Use a try/except to "manage" this OR

  • User an assert to simply:

    • Print a message OR

    • Raise an appropriate higher-level error

• What if ... and this is what comprehensive testing is- thinking about what could happen.

Manual testing

Here we can also use a separate file, but a lot more of our code is hand written. An example here is the lettery program:

# import all of the functions, comma separated.

from lotta import gen_drum, get_ticket

# run this code

if __name__ == "__main__":

print("\tTesting Drum Generator")

print(gen_drum())

print(gen_drum(10))

print(gen_drum(20))

print("\n\tTesting get ticket")

print(get_ticket())

print(get_ticket(3))

print(get_ticket(7))

Here the checking for gen_drum is visual and the testing of the get_ticket function requires the user to enter data for the test to complete. Here is the output of the above test for get_ticket (only).

Testing get ticket

Complete your ticket by entering letters!

Enter char: e

Enter char: rt

Only ONE char please!

Enter char: s

Enter char: h

Enter char:

Only ONE char please!

Enter char: @

Only ONE char please!

Enter char: ]

Only ONE char please!

Enter char: q

Enter char: e

E was already entered

Enter char: h

H was already entered

Enter char: v

['E', 'S', 'H', 'Q', 'V']

Testing Types

These are typically an exam question but also make an appearance in practical use and assignments. The types that are red include the main types of testing that you may use in your assignments.

Black box testing – Internal system design is not considered in this type of testing. Tests are based on requirements and functionality.

White box testing – This testing is based on knowledge of the internal logic of an application’s code. Also known as Glass box Testing. Internal software and code working should be known for this type of testing. Tests are based on coverage of code statements, branches, paths, conditions.

Unit testing – Testing of individual software components or modules.. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. may require developing test driver modules or test harnesses.

Incremental integration testing – Bottom up approach for testing, i.e. continuous testing of an application as new functionality is added; Application functionality and modules should be independent enough to test separately. done by programmers or by testers.

Integration testing – Testing of integrated modules to verify combined functionality after integration. Modules are typically code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.

Functional testing – This type of testing ignores the internal parts and focus on the output is as per requirement or not. Black-box type testing geared to functional requirements of an application.

System testing – Entire system is tested as per the requirements. Black-box type testing that is based on overall requirements specifications, covers all combined parts of a system.

End-to-end testing – Similar to system testing, involves testing of a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate.

Sanity testing – Testing to determine if a new software version is performing well enough to accept it for a major testing effort. If application is crashing for initial use then system is not stable enough for further testing and build or application is assigned to fix.

Regression testing – Testing the application as a whole for the modification in any module or functionality. Difficult to cover all the system in regression testing so typically automation tools are used for these testing types.

Acceptance testing -Normally this type of testing is done to verify if system meets the customer specified requirements. User or customer do this testing to determine whether to accept application.

Load testing – Its a performance testing to check system behaviour under load. Testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system’s response time degrades or fails.

Stress testing – System is stressed beyond its specifications to check how and when it fails. Performed under heavy load like putting large number beyond storage capacity, complex database queries, continuous input to system or database load.

Performance testing – Term often used interchangeably with ‘stress’ and ‘load’ testing. To check whether system meets performance requirements. Used different performance and load tools to do this.

Usability testing – User-friendliness check.. Application flow is tested, Can new user understand the application easily, Proper help documented whenever user stuck at any point. Basically system navigation is checked in this testing.

Install/uninstall testing – Tested for full, partial, or upgrade install/uninstall processes on different operating systems under different hardware, software environment.

Recovery testing – Testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.

Security testing – Can system be penetrated by any hacking way. Testing how well the system protects against unauthorized internal or external access. Checked if system, database is safe from external attacks.

Compatibility testing – Testing how well software performs in a particular hardware/software/operating system/network environment and different combination s of above.

Comparison testing – Comparison of product strengths and weaknesses with previous versions or other similar products.

Alpha testing – In house virtual user environment can be created for this type of testing. Testing is done at the end of development. Still minor design changes may be made as a result of such testing.

Beta testing – Testing typically done by end-users or others. Final testing before releasing application for commercial purpose.

Testing to "Handle" errors or flag them

The following is a brief overview of concepts covered in another section- namely handling an error using a try/except, or raising an error. We can raise an error using a try/except or using an assert statement. The following