In all cases, the test plan must address all aspects of the system. The evidence of testing should be fully documented and contain descriptions of how each test from the test plan demonstrates the reliability and robustness of the system. This should be backed up with representative samples of testing and test runs (screenshot or video evidence).
Evidence showing that the system works must be presented to enable the assessor to understand how many of the objectives have been achieved when marking the programmed solution section.
For each test carried out, it is expected that the student will describe and comment on the outcome of the test. This could be achieved by a test plan or by writing a commentary alongside the screenshots obtained that show test evidence. A suitable explanation of a test would include:
its purpose if not self-evident
the test data used
the expected test outcome
the actual outcome with a sample of the evidence, for example screen shots of before and after the test, etc, sampled in order to limit volume.
Testing (8 marks)
It is important that the testing you carry out on your solution shows that your solution both works and is robust.
You may choose to undertake the testing as you develop the program, during or at the end of each iteration. Equally, testing can be undertaken once the program is completely finished.
Regardless of when you complete the tests, you should make sure to include evidence of testing in the Testing section of your report, or at least to other previous pages that have evidence of tests. There is no need to repeat tests that you have carried out earlier.
Any of the following may be used for evidence of testing:
Format test tables showing inputs and outputs.
Video evidence of the program working with different inputs/scenarios.
Screenshots with annotation of parts of the program working.
References to other sections that show testing during iterative development.
Be aware that the testing section of your report is only worth just over 10% of the total marks, but some of the evidence that you provide of testing could demonstrate evidence for your Technical Solution.
When you carry out the tests it is essential that you focus on the core requirements for the solution. If you are providing evidence of parts of the program that are not requirements for the solution keep this to a minimum as it is not required.
When testing it is important that you select representative samples of tests and test data that will demonstrate the key parts of your program working.
Testing iteratively can be done at anytime at any point during an iteration and does not always have to be after you have completed the code.
Remember that when you compile and run the program you are already carrying out testing. At this stage you should record what has been tested and check appropriate tests at this stage of the development to make sure all scenarios are checked.
Failed tests show solid development strategy and understanding. No one writes perfect code for a complex piece of software on their first try. The exam board expect there to be errors that you will need to correct during the time completing your project.
During iterative development there is no need to repeatedly test the same thing unless you think that a change somewhere else in the code could effect the test results.
Strong iterative testing will include the following:
Summarise the tests that worked using a range of suitable evidence.
Demonstrate a few of the tests which have passed, focusing on the more interesting ones or the ones that show the system has met a core objective.
Discuss the reason why non-trivial tests failed, showing a clear line of thought to a solution.
Prove the error has been fixed.
Once you have completed your program you will need to make sure that it is fully tested. This testing will look at the system as a whole.
Remember that most of the program such as modules, classes and algorithms may have already been tested as part of the iterative testing. The focus of post-development testing is to show that everything works together as one robust program.
The tests that are undertaken at this time will probably be similar to how and end user would use the system. You will need to think about the user would use the system and mimic this use.
You may want to copy the main objectives into the the section and then design suitable tests to check that these objectives have been met.
Strong post-development testing will include the following:
Avoid repeating any of the iterative tests that have already been carried out.
Focus on the system as a whole, and not individual parts (unless the core requirements require this part to be tested and it has not previously been tested).
Mimic real-life interactions.
Directly link to the original requirements in the objectives in the design section.
You should look at employing a a mixture of the following to evidence your testing.
Tables can be a very good way of recording test evidence. These can easily be based on the objectives and are a methodical way of showing both the robustness and completeness of your project.
Comments can be added to tests where they demonstrate a line of thought when tests fail and they will show that you are considering what is causing the error.
It is advised to use a video to evidence your solution because this can quickly demonstrate that your project is robust and that it works. Video enables you to demonstrate something quickly and easily that would require a number of screenshots.
Video also enables you to easily show movement in games or transitions through a complex system. It can also record sound which means that you are able to describe what you are demonstrating so that the viewer can gauge its relevance to the projects objectives.
The length of the video produced needs to be long enough to demonstrate that the system is fully working, but you need to avoid endless trivial tests or making your teacher and moderator have to sit through a video that is unnecessarily long. Five to ten minutes should be adequate to show that your project meets the objectives set and also demonstrate the program is robust. If your solution requires a longer video then you should use more time.
Make sure you plan the tests that you will need to demonstrate beforehand so that you can be concise and cover all the important tests you want to show. If you are going to speak to explain what you are demonstrating you may find it easier to prepare a script or the voice over could be added later in editing software.
If you are providing evidence in a video you will still need to complete test tables, but these will refer to a time in the video.
Benefits
Shows transition and movement
Very suited to games
Can save a lot of time over screenshot evidence
Screen recording software can be free
Drawbacks
Can generate large files
Harder to reference
Must timestamp every reference
Not 'in the document' so must be clear where you are using the video as evidence
Screenshots are the easiest method of evidence recording. They can be created without any need for additional software and they allow you to evidence parts of your program working and demonstrate that data and instructions inputted has the desired effect.
Remember that you could also use the snipping tool on a Windows computer to select areas of the screen you want to capture.
There are drawbacks to screenshots. The most overlooked one is that your document file size will increase significantly. Screenshots can also cause a number of issues with formatting in lots of word processing applications, so that you will need to be careful that when you edit the document you check the images are still in the right place.
You will also need to make sure that any screenshots that you use are clear and easy to read by your teacher and the moderator. Check that the resolution of the image makes it clear and if there is any text that you want to be able to be read it needs to be big enough to read.
Benefits
Quick to make
Do not require special software
Good at capturing development over time
Drawbacks
Usually needs cropping
You will need to consider word wrapping and formatting
Potential for poor resolution
Shrinking screenshots can leave them unreadable
Taking too many screenshots will slow you down
Increases file size of document
Taking photos with a smart phone or digital camera can also be an option for evidencing you solution. This option will probably be more appropriate if you are working with physical computing solutions. For example, a photograph of the components put together to form the hardware for a project would demonstrate how the hardware has been put together.
Photos will need to be cropped and transferred to the report for them to be included.
Benefits
Easy to use
Do not require special software
Can get good quality from smartphones
Good a capturing physical computing development over time
Generally easy to transfer
Drawbacks
Usually need cropping/editing
File sizes from smartphones tend to be large.
You will need to consider word wrapping and formatting
Shrinking photos of screens can leave text unreadable
makes document file sizes large
Detailed design of test plan
Explain how you tested your project giving full details.
Explain how you set up your test table.
Explain your use of normal, boundary and erroneous data in your test plan.
Make your test table exhaustive, but not the screenshots / video evidence.
You could also explain how you tested as you went through (eg black box, white box, dry run, sand box, integration testing etc) giving full details of what you did.
Add the evidence of the test carried out in an appropriate manner for the project being completed.
With some projects, it may be that video is the only significant evidence that the problem is solved and that the code actually works as expected. In such cases, the student must present a full test plan that makes clear reference to video evidence. The location in the video of evidence for each test should be clearly referenced in the test plan. It is suggested that any video evidence is provided from either an uploaded video to YouTube of Google Drive, with the URL provided in the report.
Show evidence of thorough testing of your solution/investigation
Show evidence that all the core requirements of the solution/investigation have been achieved
Select representative samples of testing
Demonstrate that the solution is robust, or the investigation is thorough
If appropriate, refer to iterative tests carried out previously in the report
If appropriate, make a video of tests carried out and reference how to access it, including the time stamp in the report
If appropriate, take screenshots of testing
If appropriate, create test tables of your testing