Kafka
We have all studied producer consumer or publisher subscriber pattern, also sometimes called as pub-sub pattern.
Kafka makes things easier to implement this pattern at scale when multitudes of publishers and subscribers can be involved performing their respective jobs in order to accomplish the desired objective
With Kafka, we can expose a Topic ( queue in simple sense ) on a specefic server to publisher and subscriber which may be on same or different servers.
Producers will push events to topic which are picked up by consumers in that order. Kafka is highly scalable, fault tolerent and a robust method for streaming events. All this is done in real time
Below is an example of Kafka being used at scale in a big system at Uber
