2021-09-01 OCT

FAIR challenge: how and what can be operated with range requests

If the remote store can only be expected to have a web server, what data structures can still be operated remotely. <-- test

Jeya Balaji Balasubramanian @jeyabbalas 21:24

Both HDF5 and Zarr are suitable for range requests. HDF5 directories are stored as a single file but Zarr is stored as a classical directory structure. The file chunks are hidden in HDF5 but are explicitly visible in Zarr. Zarr is faster because of this explicit chunking. My personal preference for Zarr is both speed of read/write and that Zarr is not prone to file corruption like HDF5 is.

___

ok, let's put it to the test

• WSI formats (scale pyramids)

• VCF files (single array)

• HDF5 and ZAAR - seeing is believing :-D

___

if there is time ... ideas for post-segmentation de-arraying.