Skyhook stores its data partitions in Ceph objects, and we leverage "programmable storage"1, 2, by developing extensions to Ceph object interfaces ("cls") with data processing, indexing, and repartitioning functions that can be performed directly within the storage system. SkyhookDB provides elasticity through both the storage system's inherent ability to dynamically add and remove data storage nodes in the cluster as well as the ability to dynamically repartition data directly within storage. Because storage can perform some processing, adding more nodes to the storage cluster provides increased processing power, enhancing both IO and Compute scalability.
Our custom storage extensions include functions for data and metadata management capabilities that enable elasticity and data processing. A key benefit of programmable storage is that functions can be customized for application specific processing. Although we have started with SQL processing, the methods and approach we are developing may also be useful for other applications where in-storage processing is beneficial.
Skyhook has a 3-layer architecture, with the database on top, the external data access mechanism in the middle, and the storage layer on the bottom providing a clean separation of concerns and trust. We instantiate these layers with PostgreSQL database, PostgreSQL foreign data wrappers, and Ceph distributed object storage respectively. Each of these components are well known, mature, and trusted technologies which Skyhook builds upon and extends. This architecture is depicted in the below figure.
Each layer has a separate open source repository as noted, which are under development. Within the storage layer, Ceph Object Storage Devices (OSDs) store a collection of objects, which in our case, each object contains a database partition. Ceph OSDs have two interesting properties that we develop for our database needs.
First, Ceph objects are classes that are extensible with user defined code. For SkyhookDB, we are developing a tabular data object class that includes functions such as filtering, repartitioning (object split/merge), and data indexing. Repartitioning is more abstract but can help with both elasticity and load balancing. Additionally, more complex user defined functions or transforms are also possible. While these all represent the class of object-based operations, further functionality can be added across objects such as batching and collection level indexing by composing and extending internal Ceph components into application specific services via programmability1.
Second, each OSD has an internal indexing capability via an embedded Key-Value store (RocksDB) that is local to the device. This typically contains metadata regarding the collection of objects stored on that device. Additional metadata can be added to the KV store and queried at runtime, which can include indexing or other metadata such as counts or statistics related to object contents.
Object classes that are extensible with our user defined functions together with rich metadata storage and querying capability with the embedded KV store enable Skyhook to customize and leverage the storage system capabilities to improve scalabilty for data management .