File transfer basics for developers

Post date: Dec 9, 2013 4:28:22 PM

All this should be very clear for everyone, but it unfortunately isn't. I have seen so many broken file transfer implementations. Even if it's not hard at all. I simply hate when things need to be manually fixed, because software written to take care of the task is so bad that it gets stuck with different simple things. I also hate programs which leave junk files everywhere which require manual purging every now and then.

• If transaction is already running go to connect remote host.

• Create temp filename

• Start local transaction containing source, temp and destination filenames and sizes

• Connect remote host

• Check if final filename exists locally check size

• If local file doesn't exist go to finish transaction

• Check if final filename exists remotely check size

• If destination file exists is right size go to rename ... step

• Check if remote temp file exists, check size

• If file is smaller than local file try to resume transfer if possible

• If file size matches, good, go to Temp file ...

• If resume is not possible, delete destination temp file and re-upload whole file

• Temp file transfer is complete

• Check temp file size

• If there's mismatch retry

• If size matches and you want to be extra sure and protocol allows you to do it, check local and remote file hash at this point

• Rename temp file to it's final destination filename

• Rename, delete, move, local file, what ever needs to be done when file had been successfully transferred

• Finish file transfer transaction

• Daily or so, purge temp files from destination which are older than N days, hours or so.

Of course if these things are running in parallel, you can use the transaction table to lock files so that multiple threads / processes aren't handling same file.