Software Defined Resources
Cryptographic File Systems
Deduplication -
lessfs & btier - lessfs – A high performance inline data deduplicating filesystem for Linux. lessfs is released under the GNU GPLv3 license and can be downloaded from:
The Preudo Random Bit Bucket: Part 1 - paper 2013 [In the open source space we have filesystems like OpenDedup, Lessfs, S3QL, ZFS etc that provide deduplication even for primary online storage. While that is a desirable feature in itself, these software lack many of the advanced features of commercial products like Sepaton, HP StoreOnce or EMC DataDomain. Pcompress implements a bunch of those advanced algorithms today (I am writing a couple of papers on this) so it makes sense to extend the software into a proper scalable archival store for backup requirements. In this topic it is worthwhile to take note of eXdupe which provides archival deduplicated backup capabilities but it is quite simplistic providing only differential storage against a single initial backup dataset. It is much like a full backup followed by incremental backups. Just that there is no real multi-file dedupe. One can only dedupe the latest backup data against the first non-differential backup data. It is not a scalable chunk store that can chunk any incoming dataset and store only the unique chunks.]
Pcompress - parallel compression and deduplication utility [Pcompress is an attempt to revisit Data Compression using unique combinations of existing and some new techniques. Both high compression ratio and performance are key goals along with the ability to leverage all the cores on a multi-core CPU. It also aims to bring to the table scalable, high-throughput Global Deduplication of archival storage. The deduplication capability is also available for single-file compression modes providing very interesting capabilities. Other projects providing some of these features include Lrzip, eXdupe. Full archivers providing some of the similar features include the excellent FreeArc and PeaZIP. Pcompress is not an archiver but provides a unique combination of features to both maximize compression ratio and provide high speed.]
The main features of Opendedup are:
Variable and Fixed Block Deduplication - Efficiently deduplicate structured and unstructured data.
Reduced Storage Utilization - Deduplication can reduce storage utilization by up to 90%-95%
Scalability - Opendedup can dedup huge amounts of data and loadbalance across up to 126 independent nodes
High Availabiltiy - All block data can is replicated across up to 7 Block Storage nodes
Replication - Efficient deduplication aware replication
VMWare support - Work with vms - can dedup at 4k block sized. This is required to dedup Virtual Machines effectively
Flexible storage - deduplicated data can be stored locally, on the network across multiple nodes, or in the cloud.
Global Inline Deduplication - Opendedup shares unique data between volumes in real time.
Unlimited File and Folder Snapshot support - Support for file or folder level snapshots.
Standalone or Clustered Configuration.
Storage Developer Conference - SDC brings together solution-seeking developers, engineers, architects, product/program managers, and technical marketing managers. It will also provide managers, directors, and C-level executives the opportunity to meet the industry’s leading experts and solution-providing vendors since 1998.
Data Compression -
Node.js compression middleware. The following compression codings are supported:
deflate
gzip
btrfs compression - Btrfs supports transparent compression, meaning every file on the partition is automatically compressed. This not only reduces the size of files, but also improves performance, in particular if using the lzo algorithm, in some specific use cases (e.g. single thread with heavy file IO), while obviously harming performance on other cases (e.g. multithreaded and/or cpu intensive tasks with large file IO).
Companies:
Software Defined Storage (SDS)
Software Defined Networking
© 2009-2024
Andrey Ye. Shevel