Testing the software solution

comparison of the solution with the original design specifications

• The program must run in the correct manner, and it is crucial it runs in the manner that was outlined in the defining and planning stages, and that it has met design specifications.

generating relevant test data for complex solutions

• Test data is input used to ensure the correctness of an algorithm or program. It should be created during the analysis and defining stages, and it generally encompasses the testing of:

  • Legal and expected values: Used to demonstrate the solution produces the right output. Boundary values and those within this range, as well as critical values should be tested.

  • Illegal but expected values: Caused by ignorance or typing errors, they need to be tested and trapped so they don’t cause errors during run-time.

  • Legal but unexpected values: Needed to test whether acceptable data in an incorrect format produces the right results.

levels of testing

• Test data must be sufficient to ensure the program is completely operational and free from logic errors.

  • Unit or module testing is an effective means of testing complex solutions too difficult to test as a whole. Testing is made easier when each module performs a simple task, and that the program itself is well structured. Modules can be tested as a black box where input and output are central concerns, or as a white box where processing is the only concern. A driver program may be developed to test modules by calling subprograms or supplying variable values.

  • Program testing involves connecting modules to the main program and progressively testing the modules, mainline, and their interfaces continue to function as required. The user interface should also be tested at this time.

System testing involves testing the program on a variety of computers with different operating environments (hardware, software, OS) to detect errors.

the use of live test data to test the complete solution:

• Live data is real data, supplied by the client, that will be used on the new system. It is by using the new system as it is intended that bugs will often be detected.

• Larger file sizes may prove impossible or inefficient to open or write to, as opposed to smaller files.

• Mix of transaction types refers to the sequence of data entry or functions, and how this may affect processing. A variety of different approaches to using the program should be tested.

• Response times should be tested under heavy loads and light loads. Delayed responses might require user feedback to indicate processing is occurring, or possible code modification to improve efficiency.

• Volume data refers to the amount of data to be processed. A variety of data amounts needs to be tested to see how the program responds. Stress testing using large data volumes is a good way of checking program efficiency and robustness.

• Interfaces between modules must be tested to ensure data is correctly transferred to the module, and also that data and control is transferred back to part of the program that called it. Interface tests ensure the correct number of parameters is sent to and from the module and that the format (data type and order) is correct. They also detect conflicts between local and global variables.

• Comparison with program test data involves the generation of test data to test the limits of a program that may not be tested under “normal” use.

benchmarking

• Benchmarking is used to test a program in relation to expected outcomes and in comparison to similar products, or an established set of standards. It indicates the quality and performance expectations, and determines “real” improvements.

quality assurance

• Quality assurance is a set of procedures used to certify that a product meets specific criteria with respect to quality and reliability. Quality control involves periodic inspections, reviews, and tests on the system, relating to:

  • consistency

  • security

  • documentation

  • performance

  • recovery (from errors)

  • compatibility

  • reliability

  • stress resistance

In alpha testing, the client uses the system in a controlled environment to see if it meets requirements and records and problems, all under the watch of the programmer. Following this, the program may enter beta testing, where it is given to a number of potential users and used under normal working conditions, and they also detail any problems. After beta testing, it should be ready for acceptance.