As supercomputing welcomes new workflows of modeling, simulation, data science and artificial intelligence in the Exascale era, the goal of this site is to pose, engage, debate, and address the question - "How should the supercomputing community evolve performance benchmarks?".
The traditional benchmarking approach to engineer performance by design, i.e., building supercomputers within power and cost budgets to perform “as many” floating point operations a second has served as a directional compass for many years. It is time to rethink this approach because existing benchmarks are:
Not as inclusive/representative/comprehensive of the emerging use-cases in data-science and artificial intelligence (that need more data throughput, bandwidth, memory capacity, etc.)
Driving vendors and architects to design bespoke compute architectures that do not address a broader community interest.
Leading to proliferation of community-specific benchmarks (HPL, HPCG, MLPerf, etc.)
Curbing creativity for better processor architectures (mixed precision arithmetic, data, model and tensor parallelism, etc.)
Losing relevance that they are no-more considered competitive or worthwhile (e.g., organizations choosing not to submit results for Top500 or MLPerf)
Let us rethink and co-design how we challenge the scientists, system architects and vendors with realistic and representative benchmarks to measure and optimize end-to-end workflow performance.