IRIS

IRIS: I/O Redirection via Integrated Storage

There is an ocean of available storage solutions in modern high-performance and distributed systems. These solutions consist of Parallel File Systems (PFS) for the more traditional high-performance computing (HPC) systems and of Object Stores for emerging cloud environments. More often than not, these storage solutions are tied to specific APIs and data models and thus, bind developers, applications, and entire computing facilities to using certain interfaces. Each storage system is designed and optimized for certain applications but does not perform well for others. Furthermore, modern applications have become more and more complex consisting of a collection of phases with different computation and I/O requirements. In this paper, we propose a unified storage access system, called IRIS (i.e., I/O Redirection via Integrated Storage). IRIS enables unified data access and seamlessly bridges the semantic gap between file systems and object stores. With IRIS, emerging High-Performance Data Analytics software has capable and diverse I/O support. IRIS can bring us closer to the convergence of HPC and Cloud environments by combining the best storage subsystems from both worlds. Experimental results show that IRIS can grant more than 7x improvement in performance than existing solutions.

Design and Architecture

Features:

  • Middleware library
  • Seamless integration to applications (link to IRIS)
  • Currently supports:
    • POSIX and MPI-IO
    • HDF5 and pNetCDF
    • S3 and Openstack Swift
    • MongoDB and Hyperdex
  • Tunable data consistency
  • Relaxed metadata ops
  • Caching within IRIS
  • Prefetching for faster read
  • Non-blocking I/O


Objectives:

  • Enable MPI-based applications to access and store data in an Object Store.
  • Enable HPDA-based applications to access and store data in a PFS.
  • Enable a hybrid storage access layer agnostic to files or objects.

Goal:

  • Increase productivity, performance, and resource utilization.

Evaluation results

Real workloads

CM1 simulation

7x improvement

Montage

6x improvement

WRF Simulation

7x improvement

IRIS in hybrid mode

LAMMPS

40-60% improvement

LANL anonymous app

30-50% improvement

Acknowledgement

This work was supported by the

National Science Foundation

under grants no.

CCF-1744317,

CNS-1526887,

and CNS-0751200.