Grand Challenge

Overview

The ECE297 Grand Challenge is a competition for designing and implementing a storage server with the best performance. At the end of the term, competing teams will benchmark their storage server. The team who averages the highest score will be crowned the ECE297 Grand Challenge Champion!

Requirements

In order to qualify for the competition, a storage server must support the following functions:
  • Fully functional Milestone 4 storage server: This means the storage server must support the get, set, and query functions implemented in Milestone 2 and Milestone 3, respectively, and multi-threading support for multiple clients from Milestone 4. All test cases for Milestone 4 must pass.
  • Persistence storage: The storage server must implement the Milestone 3 bonus task for supporting persisting data to disk. The storage server must successfully pass all Milestone 3 tests that relate to this functionality. You do not have to implement this feature in time for the Milestone 3 deadline to qualify for the ECE297 Grand Challenge. It is acceptable to implement persistence after Milestone 3 is over (although you will not obtain the bonus marks attached to Milestone 3, if you achieve this functionality later).
Benchmark suite

The benchmark suite consists of a single client that sends commands to the storage server and measures two metrics: Latency and throughput. This benchmark test is considered black-box testing since the instrumentation occurs externally to the storage server. The workload consists of a mix of gets, sets, and query commands. To ensure correctness of your storage server, we will also run the Milestone 4 test cases. Note that some get commands are querying non-existing keys. These are by design to test your system's ability to detect non-existing keys. You should ignore the errors and keep going.

The benchmark contains a constant called BASE. This BASE constitutes your score. The goal is to run clients such that the time necessary to run the benchmark (on every client) takes 2 minutes or less (120 seconds). The idea therefore is maximize the BASE score while staying under 120 seconds. Sum up the BASE score of all your clients.

You can obtain the benchmark suite at the following location: /cad2/ece297s/public/benchmark.tgz. Untar the archive in your source code directory and compile benchmark.c.

Setup

You must run the benchmark suite by deploying at least eight concurrent clients, located on 8 different physical machines, connected to the same storage server, located on another machine. The clients must then start executing at the same time. It is the responsibility of the team to provide the necessary scripts to deploy such a configuration. Make sure concurrency is enabled and the storage policy is set to disk. Logging can be disabled.

To run the benchmark, use the command:

> ./benchmark <host> <port> <seed>

Host and port represents the address of the server. Seed determines the workload this client will be running. The first client should run with seed of value 1, the second with value 2, etc.

Document

To submit your entry to the Grand Challenge, you must also fill up a two-page report (IEEE two-column format). This report should detail the major design decisions with regards to performance (e.g., the optimizations made to the data structures et al.). This is then followed by a performance report where you will present the score achieved using our benchmarking tool. You should include details on how you obtained this score.

Evaluation

After the documents are submitted, we will review and validate the top three (3) scores claimed. We will verify the correctness of the code and attempt to repeat the results. If the verification fails, we will take the next highest score and attempt to validate it. Once the process is over, we will announce the top three scores for the competition.

Prizes
A bonus of 2% will be awarded to the final mark of the members of the winning team. Furthermore, an award certificate will be given to the winning team.

Schedule

Please hand in your code and your report by April 18th. E-mail your submission to the Head TA e-mail address.