Integrations and file transfers

I always say connectivity is one of the most important aspects in our lives, we are all connected whether it is on Whatsapp, Facebook, Instagram, Skype for social networking or corporate communication via software's like Microsoft Teams, Cisco Jabber, Cisco WebEx, etc. The whole point of having these systems is to transfer information from one person to another, or to collaborate and use the information for productive reasons. Nowadays we just can't imagine collaborating successfully without having these tools

The reason I gave the above analogy is to make you aware of the fact that for running an IT Financial ecosystem you must deal with a lot of sharing within the organization, outside the organization, between users or between systems. As far as the files or data shared between users is concerned its fairly straight-forward and simple to understand but the complex part would be the inter communication between different internal systems and Apps.

In an organization you have different disparate systems doing their own jobs in their own areas, but these systems need to talk, need to collaborate with other systems, without having this collaboration or "integration" your Financial ecosystem cannot function. You may always need to transfer information from System A to System B in an IT ecosystem where both systems can be inhouse or one inhouse one on cloud “Hybrid-Cloud” or both on cloud, etc. “Mind you the Hybrid-Cloud is what is being adopted by all Enterprises globally nowadays”. There can be N number of scenarios and ways to do this data transfer, let us start with the most basic and then move on to the advanced ones.

Email

I guess this does not require an explanation, as everyone knows how to operate emails and how they work, if you want to share a file with another user whether that user is internal to your organization or external, you just need to attach that file in a new "composed email" and send it to the destination email address, automatically within seconds the destination user would receive the file, but! Email has limitations when it comes to file size, in most Enterprises where you have more than 10000 users the email sending size limit would be anywhere between 2 Mb to 15 Mb, this is because if you allow everyone to use more than 15 mb email attachments then surely the organizations network bandwidth is going to get choked and also the email storage and the archival storage would burst out very soon.

Cloud file hosting solutions

There are certain cloud providers who provide you storage space on their cloud which you can use to store files and folders, these storages have limited capacity based on the plan you are subscribed to, from here you can also easily share the files with either internal employees or external employees, the sharing mechanism is very simple whenever you want to share a file a URL is generated for that file download which has a time limit (URL expires once that time is passed) and that URL is sent to the recipient on email, You can have this URL secured by setting a password to open the URL or you can also use Oauth-mechanims where you are sending it from Microsoft Onedrive and user can login using his credentials of Gmail, some major players are Google drive, Microsoft Onedrive, dropbox, etc., by the way Oauth is nothing but a standard which means information captured in a specific format which 2 parties can exchange and take decisions, in this case its authentication

DRM and file Sharing

"Digital Rights Management": This kind of sharing the file purely exists for the use case when an external organization user wants to share a file to a user or employee in our organization which normally cannot be shared via email due to maybe "file size restrictions". The DRM software would make sure that transfer channel is encrypted, and the file can only be opened by the intended recipient and it also has features such as file expiry after a certain duration and it also has geographical location tracking

DRM can also be used for internal file sharing purpose.

Ok so till now we have discussed solutions where the file transfer is happening between users, now let us look at scenarios where the transfer is required to happen between systems.

SFTP: Secure File Transfer Protocol

This is the most basic way of transferring files from one system to another, there are 2 endpoints here; one is the source, and another is the destination.

· The source can be an end-user or an automated batch job which picks up a file from a particular location and connects to the SFTP server and uploads the file

· The Destination has to be a SFTP server where some kind of SFTP service is running "SFTP server is nothing, but which can act as a server on which you can transfer files to-and-fro via Upload or download mechanisms."

The file transfer would ideally happen on a secure channel such as Https.

The Use cases would be:

· Transferring file from a partner location to our company location

· Transferring file from your location to a partner location

· Internal file transfer from one server to another "Normally would not happen via SFTP."

Some issues with the SFTP solution

· It requires an SFTP software on the destination server and SFTP client software on the source server.

· It does not provide a dashboard or reports of which files got transferred successfully and which did not.

· It can’t send alerts if the file transfer gets failed.

Managed File transfer solution

If you look at the SFTP solution, in the third use case I have clearly mentioned that the internal file movement would not happen via SFTP because there are so many systems on which the internal file transfer is required, we can’t just keep installing SFTP server software on all the servers and already SFTP is not the best option for file transfer, so what you need is a robust light weight solution which can transfer file between as many servers whenever possible with scheduling, alerting and reporting mechanisms in place

The way it works is every server which is either a source or a destination server would have a file transfer agent and this agent would act as per the instructions which it gets from the central File transfer manager.