Cluster scalability

This experiment evaluates performance as we scale out the number of storage nodes. We compare client-side (at client) processing to server-side processing (in storage). Client side incurs the cost of reading all object data from the remote server and sending over the network, while server side only sends back the data that matches the condition. The condition is applied by our specialized object class in Ceph, and for the index query we utilize the local embedded RocksDB instance on each Ceph OSD for the index lookup. The time to build the indexes for each object is reported separately.

Experimental Setup

We us the TPC-H lineitem table with 1 billion rows (~140GB). The data is stored across 10,000 objects in Ceph which are distributed over all of the server machines. Each query is executed 3 times with a cold cache and the average execution time is reported. We use parallelism of 12 (IO dispatch) per server machine since previous experiments showed that most of the concurrency gain is realized with a factor of 12.

Qb: Range query with 10% selectivity:

SELECT * FROM lineitem1B WHERE l_extendedprice > 71000.0

Qd: Point query (unique row) issued with and without index:

SELECT l_extendedprice FROM lineitem1B WHERE l_orderkey=5 AND l_linenumber=3

Qf: Regex query with 10% selectivity:

SELECT * FROM lineitem1B WHERE l_comment iLIKE '%uriously%'

Hardware

1 Client machine

1--16 Server machines (Ceph OSDs)

All machines are Cloudlab c220g2 (20 cores, 160GB ram, 1xHDD (OS), 1xSSD (data storage).

Qb results below show that client side processing (blue) improves as we scale out the number of storage servers from 1 to 16, going from over 300 seconds down to about 150 seconds. This represents the benefit from scaling out the IO. Server side processing shows a much better improvement going down to below 50 seconds as we scale out. The blue v. red results highlight the difference between only scaling out the IO versus scaling out the processing along with the IO.

Qd results below show the effect of searching all of the data for a single matching row with and without an index. In this case, our index is on the primary key of the lineitem table (l_orderkey, l_linenumber). Since all of the data must be scanned in the non-index case, we observe similar trends to the above result for Qb. However, for the index case, we see the execution time drops to within a few seconds. We note that the current implementation is not optimized and creates per-object indexes such that this query actually has to do 10,000 index lookups, one per object. Further optimizations are possible such as collection-level indexing.

Qf results below show the effect of a more CPU intensive operation, in this case regex. Our regex is precompiled so as to avoid extra overhead each time, and we utilize the google/re2 regex library. Again we observe similar trends as we scale out.