Add Headings and they will appear in your table of contents.
Copying/Replication of large volume of data over WAN (for ClusterGateOrg)
RCLONE - rsync for clouds: Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Cloudfiles, Google Cloud Storage, Yandex Files.
How to transfer large amounts of data via network - list of toolkit
RedDnet - (Research and Education Data Depot network) is an NSF-funded infrastructure project designed to provide a large distributed storage facility for data intensive collaboration among the nation's researchers and educators in a wide variety of application areas. Its mission is to provide "working storage" to help manage the logistics of moving and staging large amounts of data in the wide area network, e.g. among collaborating researchers who are either trying to move data from one collaborator (person or institution) to another or who want share large data sets for limited periods of time (ranging from a few hours to a few months) while they work on it. REDDnet is not designed or intended to be a replacement for reliable archival or long term personal storage and users must make separate arrangements to insure that the data they are sharing via REDDnet's "best effort" storage is also preserved independently with stronger guarantees.
Internet2 - Internet2 is an exceptional community of U.S. and international leaders in research, academia, industry and government who create and collaborate via innovative technologies. Together, we advance national and global research and education.
ESnet Fasterdata Knowledge Base - An Expert Guide for End-to-End Performance Tuning, Tools and Techniques
Big Data transfer over the Internet - The laboratory of the Network Technologies
PHEDEX complicated data transfer system over unreliable WAN. The PhEDEx components are:
Transfer management database (TMDB), currently version is 2.
Transfer agents that manage the movement of files between sites. This also includes agents to migrate files to mass storage, to manage local mass storage stager pools and stage in files efficiently based on transfer demand, and to calculate file checksums when necessary before transfers.
Management agents, in particular the allocator agent which assigns files to destinations based on site data subscriptions, and agents to maintains file transfer topology routing information.
Tools to manage transfer requests; CMS/RefDB/PubDB specific.
Local agents for managing files locally, for instance as files arrive from a transfer request or a production farm, including any processing that needs to be done before they can be made available for transfer: massaging information, merging files, registering files into the catalogues, injecting into TMDB.
Web monitoring tools.
BBFTP -- utility for bulk data transfer. bbFTP is a file transfer software. It implements its own transfer protocol, which is optimized for large files (larger than 2GB) and secure as it does not read the password in a file and encrypts the connection information. bbFTP main features are:
Encoded username and password at connection
SSH and Certificate authentication modules
Multi-stream transfer
Big windows as defined in RFC1323
On-the-fly data compression
Automatic retry
Customizable time-outs
Transfer simulation
AFS authentication integration
RFIO interface
bbFTP is open-source software, released under the GNU General Public License. It was written by Gilles Farrache at IN2P3 Computing Center in Lyon, France.
BBCP another tool for bulk data tranfer
mirrordir (Mirrordir version 0.10.49 (International - builtin encryption) this utility to copy directory from one machine to another one. Mirrordir makes a minimal set of changes to the directory to make it identical to the directory . Mirrordir dives into subdirectories recursively and duplicates all types of files exactly.
GridFTP Grid/Globus data transfer tool. Client part is known as globus-url-copy.
NETKIT - is an environment for setting up and performing networking experiments at low cost and with little effort. It allows to "create" several virtual network devices (full-fledged routers, switches, computers, etc.) that can be easily interconnected in order to form a network on a single PC. Networking equipments are virtual but feature many of the characteristics of the real ones, including the configuration interface.
File Transfer Service – FTS
https://fts.web.cern.ch/fts/
© 2009-2025
Andrey Ye. Shevel