Announcements
Skyhook is now part of Apache Arrow on Github.
Sept 2022. "Birds of a Feather: Pathways to Enable an Open Source Ecosystem for the Skyhook Project", Carlos Maltzahn, 2022 UC Santa Cruz Open Source Symposium.
July 2022. Release v0.4.1.
Jan 2022. On October 22, 2021, three reviewers formally approved the merge of the CROSS project SkyhoodDM into the Apache Arrow mainline. SkyhookDM will be part of the Arrow 7.0.0 release. Article.
December 2021. Release v0.4.0.
October 2021. CROSS Announcement for Skyhook Merge into Arrow.
October 2021. Big news: SkyhookDM project has been merged into Apache Arrow! Congratulations and a big thank you to Jayjeet Chakraborty and all who worked toward this goal. Please read Jayjeet's blog post for more information. github blog
September 2021. Pre-print "Towards an Arrow native storage system" available on arxiv.
July 2021. Release v0.3.0!
June 2021. Congratulations and welcome to our latest OSRE and IRIS-HEP Fellow Eshan Bhargava! This project will work toward adding `pushback` functionality for rejecting offloaded tasks under certain conditions. Eshan will be co-mentored by Jianshen Liu.
May 2021. Release v0.2.0.
May 2021. Congratulations and welcome to our latest Google Summer of Code student Yash Jipkate! This project will work toward improving and updating our documentation to be on par with the latest Apache Arrow release.
April 2021. Rados Remote Reads merged into Ceph mainline! Thanks to Ken Iizawa and Fujitsu. This PR enables direct object-to-object reads, and enables a rich set of future functionality for SkyhookDM.
March 2021. First integration of SkyhookDM merged into Coffea analysis framework. Please see getting started and try the example notebook here.
March 2021. Release 0.1.1
February 2021. Please see our latest Guide for getting started with SkyhookDM.
February 2021. SkyhookDM-Arrow Release 0.1.0!
February 2021. We posted a code walkthrough video of the new Arrow Datasets integrated code by Jayjeet. Initial Coffea package PR for Skyhook and getting started guide with test notebook here.
February 2021. New "Rados Dataset" for Arrow repository/branch - please see readme for getting started using Docker to try it out!
January 2021. Jayjeet awarded an IRIS-HEP fellowship through June 2022, congratulations Jayjeet. Jayjeet will continue his work on the new integration of Skyhook with the Arrow Dataset API and a Coffea analysis package for Skyhook. Congratulations Jayjeet!
October 7, 2020. Several SkyhookDM projects presented at the CROSS Research Symposium, programmable storage workshop session.
September 23, 2020. SkyhookDM was presented at SDC 2020 in the Computational Storage session. Slides available soon.
September 22, 2020. IRIS-HEP fellow Jayjeet Chakraborty created a post about his work on deploying and benchmarking SkyhookDM, using Popper for reproducibilty.
September 18, 2020. SkyhookDM invited talk to Kioxia.
August 31, 2020. GSoC Fellow Aditi Gupta successfully completed her summer internship with CROSS through CERN-HSF and IRIS-HEP orgs. Check out her final report here.
July 2020. Linux Professional Institute interviews CROSS Director Carlos Maltzahn about the value of open source at universities.
July 2020. Congratulations and welcome to our new IRIS-HEP fellow Xiongfeng Song! A bit about Xiongfeng and his project.
June 2020. Support for Ceph Nautilus version.
June 2020. Congratulations and welcome to our new IRIS-HEP fellow Jayjeet Chakraborty! A bit about Jayjeet and his project.
June 2020. New - Ask questions on StackOverflow with the [skyhook-ceph] tag. We also have a new GitHub Issues link, please post any potential bugs there.
June 2020. We are in the progress of creating new documentation and tutorials from our previous Wiki, creating a lighter-weight repo, and updating to support Ceph Nautilus version. Thanks very much to Ivo Jimenez for helping with this. Ivo leads the Popper project for reproducibility at CROSS, please check it out.
May 2020. SkyhookDM article appears in USENIX ;login: Magazine. "SkyhookDM: Data Processing in Ceph with Programmable Storage".
May 2020. Congratulations and welcome to our GSoC 2020 student Aditi Gupta! A bit about Aditi and her project.
May 2020. SkyhookDM was selected for Google Summer of Code (GSoC) 2020, as part of CERN-HSF and IRIS-HEP organizations. project description
Mar 2020. SkyhookDM Google Summer of Code projects now posted! We are happy to participate in GSoC for a second year in a row, this year in collaboration with the IRIS-HEP software institute. Please take a look and chat with us on our gitter channel (@jlefevre).
Feb 2020. Vault'20 presentation and pdf here.
Dec 2019. SkyhookDM to appear at Vault'20! "Scaling Databases and File APIs with Programmable Ceph Object Storage".
Nov 2019. SkyhookDM presented at the PDSW19 (co-located with Supercomputing) in Denver, CO. "Towards Physical Design Management in Storage Systems". pdf.
Nov 2019. SkyhookDM presented at 24th International Conference on Computing in High Energy and Nuclear Physics in Adelaide, Australia. "Mapping Scientific Datasets to Programmable Object Storage".
Oct 2019. SkyhookDM presented at the National Diversity in STEM Conference (SACNAS) in Honolulu, Hawai'i. "Helping Scientists Fly Over Data without Getting Swamped".
Oct 2019. Ashay presents his GSoC project and Jeff presents SkyhookDM during the storage systems track at the 4th Annual CROSS Symposium in Santa Cruz, CA.
Aug 2019. GSoC project successfully completed, please checkout the report here. Ashay embedded column-oriented data formats and processing with Apache Arrow within our SkyhookDM Ceph extensions. Thank you for a great project Ashay.
May 2019. Skyhook welcomes our first GSoC student, Ashay Shirwadkar!
March 2019. Skyhook has potential projects posted for Google Summer of Code 2019! CROSS was accepted again this year as a GSoC mentor organization and has other projects posted as well, please take a look. And see our Gitter channel to interact directly with mentors. Applications are open until April 9, 2019. More info here.
Feb 2019. Skyhook presented at Vault'19! The Linux Storage and File Systems Conference (co-located with FAST'19) in Boston, MA. (pdf slides)
Oct 2018. Skyhook presented at 2 sessions of CROSS symposium.
Sept 2018. Skyhook invited talk to Huawei analytic database group.
Aug 2018. Skyhook invited talk to Huawei storage group.
Sept 2017. SkyhookDB presented at lightning talk session of PostgresOpen 2017.
