The Blog will let you know about the Kafka i.e. One of the Frameworks of Hadoop. Kafka will be covered in some sections so that it makes easier for you to get better understanding of it. The following section will let you know about what exactly Kafka is and what was the problem that Kafka came into existence.
Why Kafka??
The above picture shows the Data Integration Problem in a Big Organisation. In any big organisation there are many source systems and multiple destination systems. The source systems act as data providers (which provides data) and destination systems acts as data receivers (which receives data). Now there is need to establish a system through which the data can flow from many source systems to multiple destination systems. One option is to create a data pipeline to move data among these systems but for a growing organisation the number of source and destination systems will be keep on increasing and you can still develop a data-pipeline for those systems but you will find that you will end up with making a too much complex data-pipeline and there will be 100% chances that some part of that data-pipeline will keep breaking everyday.
Now , the team of Linked-in discover an idea to create messaging system in order to solve such kind of problems. They have tried existing messaging systems but none of them was able to fulfill the desired requirement. Finally, they ended up with making Kafka. They developed Kafka in 2010 and Kafka gets open source in 2011. The following Figure shows you the solution of above Data Integration Problem through Kafka. The Kafka will acts as medium which stores the data coming from many sources and deliver the data to multiple destinations.
What is Kafka??
Kafka is basically a Message Distributed System . A typical messaging system have three components. Producer , Consumer and Broker. Producer is an application which is responsible for sending messages to Message Broker, Message Broker will store those messages and Consumers are applications which receives messages from Broker. Kafka also behaves in the same way but the term Message Broker in Kafka is known as Kafka Server. The following picture will give you a glimpse of Message Distributed System.
Hope Now you have an good idea of Kafka.Kafka Core Concepts and Terminologies along with its architecture will be discussed in upcoming sections.
