Change Log

• [Jan 11th] Added Resources that were posted on Piazza, but not present in the specification.

Test-driven development (TDD) is one modern technique for building software. As the name suggests, engineers write tests for every requirement in the specification before they create the implementation. This makes it easier to ensure that the final product has a base level of testing.

Adopting TDD for the course project will help ensure that you understand all the requirements of the specification before getting buried in the details of your implementation. This is important because implementing code that doesn't meet the requirements will increase the amount of work you need to do for the project.

For Checkpoint 0, you will use Test Driven Development to build a test suite against the insightUBC Section's Specification.

3. Developing Your Test Suite

After your repository has been bootstrapped, you are ready to start building your test suite for the project! You will be writing unit tests for the four methods of the InsightFacade class. The tests you write for c0 must all be contained within the file test/controller/InsightFacade.spec.ts , this is what AutoTest will use to run your test suite. Ensure that your tests do not rely on your underlying implementation (aka reference any code in your src directory, as your src directory will be replaced).

Your goal is to build a test suite that completely covers the insightUBC Section's Specification. Now would be a great time to read that specification! Please review the Section Specification Page before continuing. 

3.1 Testing framework

We will be using the Mocha Test Environment with Chai Expectations for testing. 

Mocha is a framework for running your tests and provides a structure for your test suite, defining methods such as Before, describe and it. Whereas Chai is an assertion library that provides the assert statements to determine whether a test has passed or failed (like expect). 

Some examples for testing async methods are included in the async cookbook. Three different ways to write async tests (plain chai, chai-as-promised and await) that do the same thing (resolving, rejecting and chaining) are shown below (ideally, you would pick one style of test and use it consistently in your test suite):

describe("Different ways to do the same thing", () => {

   before(() => {

       calculator = new calculator();

   })

   describe("Resolving", () => {

       // this is the most reliable way to not have test errors

       it("await", async function () {

           const result = await calculator.add([1,1]);

           expect(result).to.equal(2);

       });

       // promise approach

       it("with promises", function () {

           return calculator.add([1,1])

               .then((res) => expect(res).to.equal(2));

       });

       // 'eventually' works, but can be error prone and is not recommended

       it("chai-as-promised", function () {

           const result = calculator.add([1,1]);

           return expect(result).to.eventually.equal(2);

       });

   });

   describe("Chaining", () => {

      // this is the most reliable way to not have test errors

       it("await", async function () {

           const result1 = await calculator.add([1,1]);

           const result2 = await calculator.add([2,2]);

           expect(result1).to.equal(2);

           expect(result2).to.equal(4);

       });

       // promise approach

       it("plain chai", function () {

           return calculator.add([1,1])

               .then((res) => {

                   expect(res).to.equal(2);

                   return calculator.add([2,2]);

               })

               .then((res) => expect(res).to.equal(4));

       });

       // 'eventually' works, but can be error prone and is not recommended

       it("chai-as-promised", function () {

           const result = calculator.add([1,1])

               .then(() => calculator.add([2,2]));

           return expect(result).to.eventually.equal(4);

       });

   });

   describe("Rejecting", () => {

        // this is the most reliable way to not have test errors
        it("await", async function () {

           try {

               await calculator.add([]);

               expect.fail("Should have rejected!"); // fail if no error!

           } catch (err) {

               expect(err).to.be.an.instanceOf(TooSimple);

           }

       });

        // promise approach
        it("plain chai", function () {

           return calculator.add([1001,1])

               .then((res) => {

                   throw new Error(`Resolved with: ${res}`)

               })

               .catch((err) => {

                   expect(err).to.be.an.instanceOf(TooLarge);

               });

       });

       // 'eventually' works, but can be error prone and is not recommended

       it("chai-as-promised", function () {

           const rPromise = calculator.add([1001,1]);

           return expect(rPromise).to.eventually.be.rejectedWith(TooLarge);

       });

   });

});

3.2 My tests are failing locally, help!

Since InsightFacade is not yet implemented, any tests you add should FAIL! 

More specifically, your tests should fail when you run them locally against an invalid implementation of InsightFacade. Note that in this project, tests pass by default if you don't include or check assertions in your tests. Therefore, you must make sure you have explicitly defined asserts for every code path in every test and that the asserts check the correct conditions (according to the provided spec).

In order to improve your development iteration speed, here are a couple tips:

1. Simulate expected behaviour by temporarily modifying the InsightFacade implementation stubs to return hardcoded results (be sure to revert them before committing though!)

2. Selectively run tests by using it.only

3.3 Writing your tests

You will write tests against the four methods of the InsightFacade API. The expected behaviour of these methods is outlined in the insightUBC Section's Specification. Before writing any tests, we recommend reading through the entire specification. Then, read through the specification a second time! On your second reading, write down a list of tests you should write to cover it. Think about both the successful and error cases (when a method should resolve or reject).

All the insightFacade API methods return promises, so be careful to handle promises properly in your tests. Mocha has documentation on writing tests with promises and specifically how you need to return every promise for Mocha to be aware of it. For some tests, this will require the use of promise chaining. For example, if you wanted to add a dataset and then remove it. Checkout the Async Cookbook and the Resources section below for examples on chaining promises.

A great first test to write is the test that kills the free mutants mentioned below. 

3.3.1 Using Datasets in Tests

To test the addDataset method, you'll need to generate zip files with differing content. We recommend you place all of your test zip files in test/resources/archives. macOS users, be aware that a hidden metadata file and folder gets created in the root of the zip when you compress a folder. 

The given dataset, pair.zip, is very large and has over 60k sections! Please use it very sparingly as it will cause timeout issues. We recommend you make smaller, atomic datasets for tests that isolate behaviour.

3.3.2 AddDataset

For addDataset, the content parameter is a base64 string representation of the zip file.  To help with converting your zip files into base64 strings, we have created a utility file in /test/resources/archives/TestUtil.ts

The utility file also contains a method to remove the data directory, which should done after every test, so there is no state maintained between tests. 

3.3.2 PerformQuery

After writing down all your tests, you may have noticed there are a lot of possible tests for the performQuery method, we have created two tools to help with writing and executing those tests: the Reference UI and the a system for testing queries in bulk.

Reference UI

There is a reference UI to help with generating query test results. The reference UI is necessary to determine what the expected result is for a query. For C0 only, the order of the results from performQuery in the reference UI will be identical to the order of the results returned by our implementation. This will not be the case in future checkpoints.

Testing Queries in Bulk

When writing tests for the performQuery method, you will find that the tests have a common structure: define a query to test and the corresponding results and check that performQuery returns the correct results for the given query. To simplify this process (and to ensure that InsightFacade.spec.ts file doesn't become cluttered), we have provided starter code below for you to be able to define tests simply by writing JSON. Feel free to copy this structure to reduce the amount of boilerplate code you need to write for each new test you write for performQuery!

You can also feel free to make tests that don't use these provided structures when making your tests, for example if you think of a scenario that you don't feel is easy to test using our library.

To test queries in bulk you will need to do three things: 

1. Create query tests of the same type,

2. Organize your query tests via folders, and

3. Update your test suite to read the query tests.

1. Writing JSON Queries

In order to test your queries, your queries will need to have the same type (or JSON structure). We've defined the type below, which you can use as is, or modify to suit your needs. It should be included in your InsightFacade.spec.ts file.

/test/controller/InsightFacade.spec.ts

export interface ITestQuery {

  title?: string;         // Test case title

  input: unknown;         // Query under test

  errorExpected: boolean; // Whether the query is expected to throw an error

  expected: any;          // Expected result

}

A bit more about the type's properties:

• title - the title of the test (optional).

• input - the query you are testing. This will be the input to the performQuery function.

• errorExpected - a boolean representing if the promise returned by performQuery is expected to resolve or reject. 

• expected - the expected returned value of performQuery(input). If performQuery() is expected to resolve, expected will be of the type InsightResult[]. If performQuery() is expected to reject, expected will be a string that represents the error type.

Each query should have the same type ITestQuery.

2. Organizing the Queries

To organize your queries, you will save each query into a separate JSON file. Typescript objects roughly translate into JSON objects.

We recommend that you save your queries to subfolders within the /test/resources/queries folder. You might want to organize your queries by valid and invalid, or by other properties. For this example, we will organize our queries by valid and invalid. 

Valid Query Example:

If you've read the spec, you should recognize this query! It's the provided simple query. As you can see below, the title describes what the query is testing, the input is the query itself, the errorExpected is set to false (since it is a valid query) and the expected is the returned result.

/test/resources/queries/valid/simple.json

{

  "title": "SELECT dept, avg WHERE avg > 97",

  "input": {

     "WHERE": {

        "GT": {

           "sections_avg": 97

        }

     },

     "OPTIONS": {

        "COLUMNS": [

           "sections_dept",

           "sections_avg"

        ],

        "ORDER": "sections_avg"

     }

  },

  "errorExpected": false,

  "expected": [

     {

        "sections_dept": "math",

        "sections_avg": 97.09

     },
    ...
  ]

}

Invalid Query Example:

For this query, the errorExpected is set to true since the query is invalid! We will leave it as an exercise to determine why it is invalid. Hint: read the Section Specification. This query is expected to return in InsightError, so we've chosen to set expected to the error's name: "InsightError".

/test/resources/queries/invalid/missing_where.json

{

  "title": "Query missing WHERE",

  "input": {

     "OPTIONS": {}

  },

  "errorExpected": true,

  "expected": "InsightError"

}

3. Bulk Query Tests

Now you've got all your queries under test in an identical structure (ITestQuery), and they are organized into JSON files within subfolders of the /test/resources/queries/ directory, so it is time to start testing them! 

Since you will likely need to write many queries to fully test the performQuery specification, an important consideration when designing the structure of your test suite will be the amount of boilerplate code required. We also want to make sure that we can easily run and debug individual queries to help us quickly locate faults when we are implementing performQuery later in the project. One way to meet these requirements is the following:

it("[valid/simple.json] SELECT dept, avg WHERE avg > 97", checkQuery);

This feels relatively clean: the first string parameter includes a clear description of what the test will assert, and the square brackets specify the location of the JSON file containing the query. If the test fails, you will know immediately which JSON file to look at and the query that was executed. The second parameter is a function that will execute when the test runs. The  checkQuery function will need to read the JSON  file (based on the path specified in the test name), invoke  performQuery, and assert that it returns the result specified in the JSON file. Is this design perfect? No: the biggest downside is that we are embedding a file path in the test name which means that whoever is writing the test will need to know this is a requirement. However, having the name of the JSON file in the test output will  be helpful as you are debugging your code.

4.  Free Mutant Walkthrough

The free mutant:

The first mutant is in the API method addDataset() and it causes addDataset() to accept an empty dataset id. For example, if a valid dataset is added with the id "" and the type "sections", it will be successfully added (Oh No! :-). 

Let's kill this mutant together!

This mutant is in the addDataset API method, which is defined in the IInsightFacade.ts file as:

addDataset(id: string, content: string, kind: InsightDatasetKind): Promise<string[]>;

Let's look at each part of a test and what we will require to kill the mutant: Setup, Execution and Validation.  

• The Setup contains any code required to get your application into the desired testable state.

• The Execution contains your method under tests. 

• The Validation contains any asserts. 

Setup

Do we need our InsightFacade instance to be in any specific state for this test? What method calls on InsightFacade (like addDataset, listDatasets etc) do we require to run this test? 

For this test, we require no setup. We aren't relying on another dataset already being present or any other state for Execution, so we can move onto Execution.

Execution

For the execution, we need to call our method (addDataset)with the correct arguments. Let's look at each argument one by one.

1. id
   As mentioned in the free mutant description, we want our our id to be an empty string, so "". The empty string is considered an invalid dataset id.

2. content
   The content should be valid, since we only want this test to fail due to our invalid id, not because of any other invalid input.
   
   Looking at IInsightFacade again, it describes the content as:
   @param content  The base64 content of the dataset. This content should be in the form of a serialized zip file.

Ok, so we want the base64 representation of a valid sections dataset zip file.  A valid dataset can be found under "Valid Content argument to addDataset" in the Section Specification. Once downloaded, we recommend adding it to the test/resources/archives directory as mentioned in the "Writing Your Tests" section.

Now we need to convert this valid sections dataset into a base64 string. Also in the "Writing Your Tests", you will find the /test/resources/archives/TestUtil.ts file. We can use the helper method getContentFromArchives to convert our zip file into a base64 string. 

We will convert the zip file into a base 64 string with:
const sections = await getContentFromArchives("yourzipfile.zip");

3. kind
   The kind should be valid and since we are adding a sections dataset, it should be  InsightDatasetKind.Sections

Putting that all together, the Execution part of our test will look like:

const facade = new InsightFacade();
const sections = await getContentFromArchives("yourzipfile.zip");

const result = await facade.addDataset("", sections, InsightDatasetKind.Sections);

Validation

addDataset returns a promise, which means the result object in our Execution will be a promise. There are a variety of ways to write tests with promises. Checkout the Async Cookbook for examples.

While you can use the the chai-as-promised library, this is a frequent source of errors; using async/await in your test cases is the best way for your tests to fail as you expect. If you use chai-as-promised, you will need to set this up before the asserts will work correctly. Look at the "Installation and Setup" part of their documentation. The eventually keyword is chai-as-promised and is described in their documentation.

Mocha

Mocha is the test framework and provides the structure in which we should write our tests. In their documentation, they explain how to use Mocha with promises and the different useful methods (it, before). If we were to move our above Setup/Execution/Validation into a test, it would look like:

it ("should reject with  an empty dataset id", async function() {
    const facade = new InsightFacade();
    const sections = await getContentFromArchives("yourzipfile.zip");

    try {

        const result = await facade.addDataset("", sections, InsightDatasetKind.Sections);

        expect.fail("Should have thrown!");

    } catch(err) {

        expect(err).to.be.an.instanceOf(InsightError);

    }

});

Mocha requires all promises to be return ed. This is a very important step for handling promises. All promises must be returned so Mocha can wait for them to resolve or reject appropriately.

But... we can do much better! The facade and sections dataset will be used in other tests, so we can clean things up by using Mocha's before and beforeEach hooks. Let's also use some describes!

describe("InsightFacade", function()  {

    describe("addDataset", function() {
let sections: string;

        let facade: InsightFacade;

        before(async function() {

            sections = await getContentFromArchives("yourzipfile.zip");

        });

        beforeEach(function() {

            facade = new InsightFacade();

        });

        it ("should reject with  an empty dataset id", async function() {
            try {

                const result = await facade.addDataset("", sections, InsightDatasetKind.Sections);

expect.fail("Should have thrown!");

            } catch(err) {

                expect(result).to.be.instanceOf(InsightError);

            }

        });

    });

});

sections goes into the before because it acts like a constant and its state won't change between tests. However, facade's state will change  since we'll be adding datasets and removing them, which is why we reinstantiate it between each test.

clearDisk()

In Section 5.5 Handling Crashes, we mention how datasets will be written to disk so they can be persisted across InsightFacade instances. Let's look at the following test suite together:

describe("InsightFacade", function()  {

    describe("addDataset", function() {
let sections: string;

     let facade: InsightFacade;

     before(async function() {

            sections = await getContentFromArchives("yourzipfile.zip");

     });

     beforeEach(function() {

            facade = new InsightFacade();

     });

        it ("should successfully add a dataset (first)", async function() {
            const result = await facade.addDataset("ubc", sections, InsightDatasetKind.Sections)

            expect(result).to.have.members(["ubc"]);

        });

it ("should successfully add a dataset (second)", async function() {
            const result = await facade.addDataset("ubc", sections, InsightDatasetKind.Sections)

            return expect(result).to.have.members(["ubc"]);

        });

    });

});

Notice - we've got the same test duplicated - will they both pass? They would if there is no state carried between the tests. But, in Section 5.5, we state:

When a user creates a new instance of InsightFacade, they should be able to query the datasets existing on disk...

After the first test, the dataset "ubc" will be written to disk. When we create another instance of InsightFacade, it will see the "ubc" dataset on disk. Then, when the second test adds the dataset "ubc", it will be rejected since it can see that the dataset "ubc" has already been added. The first test will pass and the second test will fail.

We need to remove the dataset(s) from disk between tests, so each test starts with a fresh state. You may have noticed that /test/resources/archives/TestUtil.ts had an additional helper method, clearDisk(). Just like we need to reinstantiate facade between tests, we should also be clearing the disk. A call to clearDisk() should be added BEFORE creating a new InsightFacade instance. The beforeEach should look like:

beforeEach(async function() {
    await clearDisk();

    facade = new InsightFacade();

});

We'd also like to drop a big hint about the location of another mutant. In the query EBNF, the specification allows the usage of wildcards using the asterisk "*".  One of the mutants contains a defect where wildcards behave improperly in some way!

5. Resources

5.1 Getting started

Have you read the Project Overview page yet? It contains an overview of the four project checkpoints and how each part will connect to build insightUBC.

A great first step would be to ensure that all the required tools are on your machine: git, node, yarn. Then, work through the Initializing Your Repository section, using git to commit your progress. Google as always is an excellent resource, as most things used in this course are fairly popular and well documented.

If you aren't familiar with promises and asynchronous code, it's worth taking the time to learn about it through one or both of the course provided resources: Async Cookbook and/or Promises Video.

When you are ready to start developing your test suite, we recommend starting with the walk through that explains how to kill the first mutant (the walk through is below). Then follow the hints regarding where the second mutant lives. 

5.2 Recommended videos & tutorials

The following resources have been created by course staff to assist with the project.

Videos

Typescript: an introduction to TypeScript.

Promises: an introduction to promises and their asynchronous properties.

Tutorials

Git Cookbook: learn the basics of Git.

Async Cookbook: learn about promises and the differences between synchronous and asynchronous code.

6. Appendix

6.1 Injected Main.ts file

To ensure that the correct members of the API are defined and exported correctly, we will add a file to your repo which imports the API before commencing the build. This file as is follows

// Run from `src/Main.ts`

import * as fs from "fs-extra";

import * as zip from "jszip";

import {

    IInsightFacade,

    InsightDatasetKind,

    NotFoundError,

    ResultTooLargeError,

    InsightError,

    InsightDataset,

    InsightResult,

} from "./controller/IInsightFacade";

import InsightFacade from "./controller/InsightFacade";

const insightFacade: IInsightFacade = new InsightFacade();

const futureRows: Promise<InsightResult[]> = insightFacade.performQuery({});

const futureRemovedId: Promise<string> = insightFacade.removeDataset("foo");

const futureInsightDatasets: Promise<InsightDataset[]> = insightFacade.listDatasets();

const futureAddedIds: Promise<string[]> = insightFacade.addDataset("bar", "baz", InsightDatasetKind.Sections);

futureInsightDatasets.then((insightDatasets) => {

    const {id, numRows, kind} = insightDatasets[0];

});

const errors: Error[] = [

    new ResultTooLargeError("foo"),

    new NotFoundError("bar"),

    new InsightError("baz"),

];