EvoBench is a a standardized benchmark to measure and compare Schema Evolution Systems (SES) like Darwin. Using Docker containers, EvoBench is easy to set up, its design and operation ergonomic, its results easy to interpret and reproduce - even after decades.
To address the problem of continuous reproducibilty on different systems and to keep it simple to set up the necessary software stack (simplicity), containerization with Docker is used. We therefore provide images with pre-filled and configured databases and the SES to be benchmarked as an ready to run Docker image.
To keep the configuration of the benchmark as simple as possible one central configuration file in the JSON format is provided. This means that the entire configuration necessary for the execution of the benchmark can be made within this file. To be able to define different benchmark scenarios it is possible to specify several of these files.
A REST interface is used for the communication with the SES of choice (e.g. Darwin). This allows to easily extend EvoBench to support additional SESs which provide a REST API.
The following is an overview of the benchmark implementation design using Docker containers:
The measurements can be easily reproduced with a single command (the only requirement is Docker):
docker run -t --rm \
--network host \
-v "/var/run/docker.sock:/var/run/docker.sock" \
-v "$(pwd)/results:/opt/evobench/results" \
dbishagen/evobench:20210617.1-alpha \
-c ./config/experimental_setup_1.json
The Docker containers for reproducing the proof of concept measurements with our NoSQL Schema Evolution Benchmark are available at Zenodo.
André Conrad, Mark Lukas Möller, Tobias Kreiter, Jan-Christopher Mair, Meike Klettke, Uta Störl: EvoBench: Benchmarking Schema Evolution in NoSQL. TPCTC 2021: 33-49
Mark Lukas Möller, Meike Klettke, Uta Störl: EvoBench - A Framework for Benchmarking Schema Evolution in NoSQL. IEEE BigData 2020: 1974-1984