Message Queue

So, I don’t need to remind you that we started our discussion about data transfer from one system to another, we discussed SFTP, we discussed API and discussed Integration BUS and then API-Gateway. Now is the time to understand another concept called "Message Queues" which is also an exceedingly popular mechanism of data transfer based on specific Use-cases where you don’t want an immediate response from the destination system, to make it easier let me break it into parts?

· MQ uses messages to transfer data i.e., it is similar to what we just saw in the API object, you can safely imagine that object is the message or to make it simpler; imagine its similar to an email message and every message is a file or an object which can be stored in a Queue (a software pipeline where messages are stored and sequenced)

· MQs are always used for Asynchronous communication which means the source system would just send a message and does not need a response immediately on the same session.

· These Messages must be stored in an intermediary system which actually sits between the source and the destination system, this system does something called as "Store and forward" of the messages, this intermediary system is called as the Message Queue system

The intermediary system which handles these messages and Queues is called as an MQ system, so now we know that "MQ system" is a system which sits between 2 systems which want to transfer data and has a Queue i.e., a pipeline which basically keeps a track of the sequence how the messages are stored so that they can be picked up by the destination system in a FIFO (First in First out format)

The system which sends the message to the MQ server / instance is called the Publisher (a system which publishes the message)

The system which picks up the message from the message Queue is called as a subscriber which subscribes for the messages.

A single MQ server can have Multiple Queues which can be accessed by N number of Publisher / Subscriber systems to exchange data.

Many applications in the industry are heavily dependent on messages and MQs, one of the most popular System is SFMS which does the RTGS / NEFT transactions, so when you are basically making a transaction "Sending money to someone", this transaction is actually converted into a message format by your bank and sent to NPCI which in return sends it to the Bank of the Beneficiary (to whom you are sending money), this message is then interpreted by specific applications which I am calling as SFMS and then sent to the core banking system to make a credit/Debit in the Beneficiary account

Note: - One more important point is that the concept of Message queues helps reduce the load on the source and destination systems as mostly the destination system has the liberty to pick up and process the messages when its actually free on resources and cannot be overburdened as it happens in case of an API where a 1000 API calls at once can bring down your destination system as it has to process so many requests and provide the response at the same time