Harmonia

Harmonia: An Interference-Aware Dynamic I/O Scheduler for Shared Non-Volatile Burst Buffers

Harmonia Architecture

  • Schedule I/O phases instead of entire application
  • Overprovision buffer nodes
  • 5 scheduling policies to fit a variety of workloads

Modern HPC systems employ burst buffer installations to reduce the peak I/O requirements for external storage and deal with the burstiness of I/O in modern scientific applications. These I/O buffering resources are shared between multiple applications that run concurrently. This leads to severe performance degradation due to contention, a phenomenon called cross-application I/O interference. In this paper, we first explore the negative effects of interference at the burst buffer layer and we present two new metrics that can quantitatively describe the slowdown applications experience due to interference. We introduce Harmonia, a new dynamic I/O scheduler that is aware of interference, adapts to the underlying system, implements a new 2-way decision-making process and employs several scheduling policies to maximize the system efficiency and applications' performance. Our evaluation shows that Harmonia, through better I/O scheduling, can outperform by 3x existing state-of-the-art buffering management solutions and can lead to better resource utilization.

Harmonia Highlights

Harmonia, (Greek word meaning "in agreement or concord"), is a new dynamic I/O scheduler tailored for systems with shared I/O buffering nodes.

Harmonia is a scheduler that is:

  • interference-aware,
  • operates in a finer granularity, and is
  • adaptive to the current system status.

Harmonia makes scheduling decisions by collecting information from applications (i.e., intention to perform I/O) and the buffering nodes (i.e., available or busy device status) at the same time.

Harmonia uses a dynamic programming algorithm to optimize the scheduling of individual I/O phases on available buffers. By over-provisioning buffer nodes and by overlapping computation with I/O, Harmonia is able to reduce the scheduling cost in several metrics such as

  • maximum bandwidth,
  • minimum stall time,
  • fairness, and
  • buffer efficiency.

Interference Metrics

Storage Medium Sensitivity to Concurrent Access

MSCA - Write

NVMe are less sensitive to concurrent access than SSDs

MSCA - READ

Traditional spinning disks are demonstrating high MSCA values

Interference Factor

If- Write

If provides an absolute reference for a non-interfering system when If = 1

iF- READ

Context-dependent metric: Allows comparison of applications with different I/O size or I/O requirement

Harmonia Scheduling Policies

Dynamic Programming Approach. Each policy has different cost function

Evaluation Results

Testbed specs

  • Testbed: Chameleon System, Appliance: Bare Metal
  • OS: Centos 7.1, Storage: OrangeFS 2.9.6, MPI: OpenMPI
  • Programs:
    • Synthetic benchmark alternating computing and I/O phases
      • Workloads:
        • Compute-intensive
        • Balanced
        • I/O intensive
        • Only I/O
    • VPIC: particle simulation
    • HACC-IO: cosmological simulation I/O kernel
  • Buffer aggregate capacity: 800GB
  • Total dataset size for 8 instances: 1.6TB

Performance and overheads

  • 40% faster execution than DataWarp for 8 concurrent instances
  • 4% overhead on average to perform I/O phase detection offline
  • MaxBW offers the best I/O time whereas Fairness the slowest I/O
  • Harmonia’s scheduling policies offer greater flexibility to the system

Synthetic Workload

  • Synthetic benchmark
    • Balanced workload (compute-I/O)
  • Average completion time
    • Wait to be scheduled
    • Computation time
    • I/O time
    • Overheads
  • Concurrent execution scaling
    • 2-8 instances
    • Buffer can hold data up to 4 instances before they lush
  • Compared to DataWarp scheduling

Scheduling Metrics

Max Buffer Efficiency

2x more efficient

Max Bandwidth

3x higher average bandwidth

Fairness

10x higher fairness

Min Stall Time

minimize stall time by 3x

Harmonia’s policies can better adapt to workloads than other buffering systems

Harmonia Summary

  • Cross-application I/O interference is a source of performance degradation that I/O schedulers need to be aware of.
  • Buffering mediums (i.e., storage devices) handle concurrency differently which can be effectively used by the scheduler to minimize interference.
  • Policy-based scheduling works better for diverse I/O workloads.
  • Harmonia, a new, dynamic, interference-aware I/O scheduler
    • schedules individual I/O phases for finer granularity.
  • By overlapping computation and I/O phases and calculating I/O interference into its decision making process, Harmonia can be 3x faster than state of the art buffering systems leading to better resource utilization.

Acknowledgement

This work was supported by the

National Science Foundation

under grants no.

CCF-1744317,

CNS-1526887,

and CNS-0751200.