Embedded Files
Recently, as part of a project, I had the opportunity to work with Apache’s Kafka to establish communication between microservices. Before diving into the technical details, let’s first understand the concept of asynchronous communication with a simple real-life example. Imagine two scenarios of communicating with your boss. In the first scenario, you’re on a phone call, and your boss asks for an update on the day’s tasks. You’re under pressure to respond immediately with accurate facts and figures, which can be challenging if you’re not fully prepared. Now, consider a second scenario where your boss requests the updates via text or email. This approach is much more convenient for you. You have the time to gather all the necessary information, organize it properly, and present it in a well-structured manner without the stress of responding instantly. Here, communication through email represents asynchronous communication. In asynchronous communication, the sender and receiver do not need to be synchronized in real time. The sender can deliver the message, and the recipient can respond at their own convenience.
Imagine you are developing a food delivery application and have divided it into two layers. The first layer handles business validations for incoming requests, while the second layer forwards these requests to partner APIs to place orders. Now, suppose the second layer becomes temporarily unavailable due to high traffic. If the communication between these two layers is synchronous, customers would encounter “technical errors.” However, with Kafka in place, the requests from the first layer would be stored in a topic until they are processed by the second layer. This prevents technical glitches from affecting the customers. Instead, the application can display messages like “Request raised” or “Confirmation pending” on the user interface, ensuring a smoother and more user-friendly experience compared to the previous scenario where customers faced immediate errors.
Here I would explain the basic components of Kafka architecture and how you can implement asynchronous communication between two or more microservices using Kafka. Before we dive deeper, we need to be familiar with few terminologies — brokers, messages, topics, producers and consumers as these are essential components for implementing service communication using Kafka.
A Kafka server is referred to as a broker and multiple brokers are referred to as a cluster. A message is basically a record/event/request sent by one of the producers and will be consumed/read by one of the consumers. The basic responsibility of a broker is receiving and storing message from producers and facilitating the consumers to fetch the message intended to be consumed by them. Producers, as the name suggest are basically the senders in this communication scenario. They write the request/message to entities called topics which can be read/consumed by the consumers.
Multiple producers can publish to a single topic and similarly, multiple consumers can consume data from a single topic. Kafka categorizes messages into topics. Ideally, the messages in a topic should be related. For example, in a hotel management service, the topic named orders will have all orders placed. Kafka has provisions for configuring the retention policy for individual topics. This basically means that we can configure how long a particular message stays in a topic before it gets dropped. Any message stays in the topic for this configured time period irrespective of the fact that it gets consumed or not. The topics are subdivided into several partitions as depicted in fig. 1.
Whenever a producer publishes a message into a topic, it gets appended into one of its partitions depending upon the key we are passing with the message. Here the word “appended” is of high importance. It conveys the sense that order of message is maintained in the partition. When a message is published into a topic, it can be written to any of the partitions inside the topic (load balancing can be implemented here), but it will always be appended to the end of the queue in that partition. Refer to fig. 2 below for clarity.
Generally, a key is also generated with the message before the publisher publishes the message to the Kafka topic. This key is optional, but it becomes quite useful if the message is to be published into a particular partition of the topic. Ideally, a particular partition should have message with same/identical key. The keys can be configured to be of any data type: integer, string, UUID, etc.
Hopefully, we are now comfortable with the components of Kafka architecture and their primary roles in it. Now, let us understand how Kafka helps us implement asynchronous communication between microservices. The producer generates the message and publishes it to the broker. This broker then decides the partition where the message is to be published. This decision is influenced by several factors like the key generated with the request, the load balancing approach implemented, etc. Once the message is sent into the partition, now the consumers start issuing fetch requests to the brokers leading the partition it wants to consume. Logging is implemented in Kafka which helps keep track of what has been consumed. If the consumer is down or out of service due to some technical mishaps, the message resides in the topic — this facility to keep requests in the topic reduces the back pressure on the downstream systems as well which is very helpful during hours of high traffic. Once the consumer is active and resumes its services, it can consume the messages in the topic that were intended to be consumed by it. This ability to consume message as per the availability of the consumer helps in loose coupling of the microservices and facilitating independent development and deployment. The consumer and the related services are under no pressure to respond to the request immediately.
The advantages of Kafka based communication are as follows:
Loose coupling: Independent development and deployment of services.
Scalability: Kafka helps building scalable application as it can handle high volume of data and traffic. This makes it an exemplary solution for real time data processing services.
Fault tolerance: Kafka ensures message delivery and processing even case of service down-time and network failures. It also makes the service highly available.
Reduced Back pressure: Kafka implementation ensures that the back pressure is reduced on the downstream systems as well, since the request reside inside the Kafka topics until it gets consumed. So, even if the downstream APIs are unavailable, Kafka manages those scenarios as well.
These advantages of Kafka have made it an extremely demanding technology. Its ability to provide these benefits even with requirements managing huge amount of data is praise worthy. Several companies and firms like Netflix, Uber, PayPal, etc. have already implemented solutions using Kafka and several others have started migrating their architecture to Kafka. The demand for professionals having a knack for implementing Kafka based solutions is all time high and is predicted to rise even further. So, next time when you are trying to implement communication between microservices you can give Kafka a try!
Page updated
Google Sites
Report abuse