AutoTest

AutoTest is our system for evaluating your projects on GitHub, and serves as the "client" for the project when run for each checkpoint, and as a smoke test suite when during the intermediate development period.

LOCAL TEST SUITE

You are responsible for thoroughly testing your own system, as intermediate feedback on your progress will not be provided on demand.

To help you raise your confidence in your own test suite, AutoTest will run your tests against a reference implementation of the project, and those scores will be reported back to you via #c0. The reference implementation may not be perfect -- when in doubt, refer to the spec. If you're confident you've found a bug, make an issue and tag us -- we'll look into it. Confirmed bugs will be tracked in the bugs folder on Piazza.

The names of the failing tests provided that the exact string of the test name is present in your source files.

This means a test written like
it("Should" + " add a valid dataset", function () {...});
will not have its name reported back upon failure, as the test name is created dynamically. This restriction is in place to prevent grading data exfiltration.

The #c0 suite will only the test file test/controller/InsightFacade.spec.ts which tests the public API for the project. Ensure that this file does not rely on your underlying implementation, as your src/ directory will be replaced. The #c1, #c2, and #c3 suites will execute all of your tests, but only to measure their coverage of your implementation.

You are encouraged to all all the tests you need to your test/ folder, in order to comprehensively test your implementation.

SMOKE TEST SUITE

The smoke test suite is comprised of basic tests that you are likely to already have in your own test suite, and is a subset of the client suite described below. On merge commits to the main branch, you may invoke AutoTest to see your smoke test suite results.

AutoTest is not meant to be a replacement for testing your code locally with your own test suites. The smoke test suite provided is primarily to give you confidence that your code is being executed correctly on our platform.

CLIENT TEST SUITE

The AutoTest client suite contains a variety of integration tests to validate and exhaust your implementation. The client suite will be run and shown three times during the term, following checkpoint deadlines.

Submitting to AutoTest

You will be able to submit your assignment for smoke test validation by merging to main. The client suite will be run on scheduled dates throughout the term. AutoTest will return information on how you are doing on various features of your project, but not specific test details. Due to this and the rate limiting, it is highly recommended you extend your project test suite to make your code as robust as possible before merging to the main branch.

AutoTest is invoked using a bot on GitHub. If you make an @310-bot comment on a commit in GitHub the bot will be invoked. Test results will be reported back to the same commit when the test is complete. Some notes to keep in mind:

• • If your code does not cleanly compile or lint (e.g., yarn tsc && yarn lint), AutoTest will not run the rest of the commit and it will not be graded. This will not use your limited submission.

  • If a test times out it will be counted as failed. To be safe about not hitting this timeout, try and ensure no individual test takes more than about 5 seconds to run locally in your personal test suite.

  • AutoTest will run your code on all pushes, regardless of if you call it. This means that you may get messages/emails about build failures even if you don't call the bot. Sometimes when the processing queue is long, you may get emails about old pushes, this is fine and won't affect your most recent work.

  • The only timestamp AutoTest trusts is the timestamp associated with a git push event. This is the timestamp that will be used for determining whether a commit was made before a deadline (this is because commit timestamps can be modified on the client side, but push timestamps are recorded on the AutoTest server itself). Make sure you push before any deadline and know that any pushes after the deadline (even if some commits within that push appear to come from before the deadline) will not be considered.

AutoTest tips

Your AutoTest submission may fail for the following reasons:

• • Exceeding 5 minutes of total execution time (with the exception of Checkpoint 0, which can take up to 20 min and that's expected). This should not happen, but if it does it is probably because you either have an infinite loop somewhere or are not settling your promises.

  • Producing more than 5 MB of output to the standard output. It is easy to generate 100s of MB of output; please think carefully about the logging statements your code is making.

  • Depending on invalid dependencies. If your code uses libraries it shouldn't, it will fail. We do not run yarn install for your code, instead we copy the approved set of libraries to your target project. The approved list of dependencies can always be found in the package.json file in the repository we provision for you.

  • Importing modules is case sensitive in Linux. (e.g., import jszip = require('JSZip'); will work on Windows and probably OS X but for it to work on Linux you need import jszip = require('jszip');). That said, you probably shouldn't need to use require at all (e.g., import * as parse5 from "parse5"; should be all you need).

  • AutoTest will only work on the repositories we provision; it will not work on forked repos.

Submitting your deliverable

Your final grade will always be based on the maximum graded commit on main branch of your repository made prior to the final deadline.