Overview

Rsyncmanager is a daemon designed to simplify the task of coordinating multiple file transfers to or from a machine. It was originally written to serve as the managing hub of a backup system that used rsync to feed in files from many other machines, so that all backups could be run from a single place. That said, it can be used to coordinate any kind of regular file transfer, from a simple hourly copy of an email store from a primary to a backup server, to multiple concurrent rsyncs to multiple machines on varying schedules.

Features

• Coordinate multiple file transfers using rsync, scp or a combination of the two
• Limit the number of concurrently running file transfers to prevent resource starvation
• Prevent file transfers from starting based on arbitrary conditions using external scripts (time of day, machine load, memory usage, phase of the moon…)
• Run commands before and after a file transfer to handle setup and cleanup conditions (e.g. dump a database to disk, transfer it to a backup, then delete the dump)
• Intelligently limit concurrent file transfers to or from the same machine to prevent resource limitations from slowing or starving a transfer
• Limit concurrent file transfers based on arbitrary grouping (e.g. only one transfer from internal to external servers over a slow line at any given point)
• Coordinate transfers between multiple machines (grouping over multiple machines, avoiding having two machines push data to a third at the same time, etc.)
• Provides a clear web-based administrative report of all currently managed transfers and their current status (running, behind schedule, etc)
• Intelligently schedule file transfers based on average runtimes so that transfers happen within a defined schedule but do not run constantly if not needed

Requirements

• ruby 1.8.x
• rsync (if rsync is to be used for file transfers)
• scp (if scp is to be used for file transfers)

Get It

Rsyncmanager can be easily gotten as a gem through rubyforge, either by downloading it here or by using the gem command directly:

gem install rsyncmanager

A plain tarball of the latest code can be gotten here.

Configuration

Almost all configuration of rsyncmanager is handled through its XML configuration file. It can be placed anywhere on the system and is referenced with the -c commandline parameter. A very complete example follows at the end of the README included in the distribution – although many sections can be left out. A much simpler example looks something like this:

    <configuration>

            <general>
                <maxtransfers>3</maxtransfers>
                <transferdefaults>
                    <type>rsync</type>
                    <options>-aSrq --delete</options>

                </transferdefaults>
            </general>
            <transfers>
            <transfer>
                <description>disk2 groups</description>

                <frequency>10m</frequency>
                <from>/mnt/filestore/groups/</from>
                <to>172.16.1.2:/mnt/filestore/groups/</to>
            </transfer>

            <transfer>
                <description>disk2 personal</description>
                <frequency>10m</frequency>
                <from>/mnt/filestore/personal/</from>

                <to>172.16.1.2:/mnt/filestore/personal/</to>
            </transfer>
        </transfers>
    </configuration>