Hopefully, now you are familiar with the Kafka Terminologies.  The following blog will make you understand about the how to implement Kafka on your Linux terminal.  Along with implementation some other concepts of Kafka are also included in this blog.

Steps to Implement Kafka

The steps are defined as :-

1. Download & Install Kafka
2. Start Kafka Server (Broker)
3. Create a Topic
4. Start a Console Producer
5. Start a Console Consumer
6. Send and receive Messages

Note:

If you want to run Kafka in your system then you must have Linux installed on your computer. Window users can also implement Kafka on their system but for that they have to first install a Virtual Box for Linux Machine in their system. 

1. Download and Install Kafka

You have to visit the below link and download the mentioned tar file in the given link. There are  alot of versions are available for Kafka, you can choose any of them and download tar file for selected version. I have downloaded the 0.10.1.0 version of Kafka tar file.

URL= https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.0/kafka_2.10-0.10.1.0.tgz

• Go to above URL.

• Download the top-most tar file among available list of tar files present on above URL.

• Once you downloaded the tar file then un-tar it.

• Once you have untar file with you then open the folder and you will see following directories over there as shown below.

There are mainly three main Directories on which you have to put more focus. These are:

• Bin – It will be containing command line commands which you will be using in-order to implement Kafka via Linux terminal.
• Lib – This directory is responsible for containing jar files.
• Config – Config directory contains all the configuration files which are helpful in setting up the desire configurations.

2. Start a Kafka Broker

Note: Before Starting Kafka Broker(Server), It is mandatory to start Zookeeper First.

• Zookeeper :   Zookeeper is used to provide coordination services among distributed systems. Kafka is  also a distributed System so we need a system which will provide coordination services among Kafka Brokers. In simple words, zookeeper act as an intermediate between multiple brokers.  The following image will show the steps and command to start the zookeeper.
• Initially you have to go to directory where your Kafka UN-tar File is Placed.
• Enter Command (bin/zookeeper-server-start.sh  config/zookeeper.properties) to start the zookeeper as shown below in image.
• The command is executed over shell script file (zookeeper-server-start.sh) which takes one parameter as an argument i.e. (config/zookeeper.properties). It accepts the zookeeper.properties file as an argument which is placed under config directory.
• Press Enter after writing the above command over terminal

• After pressing Enter you will see the following content on screen and the last line which contains default port number of Zookeeper i.e. 2181 , ensures that the zookeeper has started successfully.

• Now it’s time to start Kafka Broker (Server).
• Open the another terminal and must ensure that you are present in same directory where the un-tar Kafka file is present.
• Enter Command (bin/kafka-server-start.sh  config/server.properties)to start the Kafka Broker as shown below in image.
• The command is executed over shell script file (kafka-server-start.sh) which takes one parameter as an argument i.e. (config/server.properties). It accepts the server.properties file as an argument which is placed under config directory.
• Press Enter after writing the above command over terminal

• After pressing Enter you will see the following content on screen and the last line over terminal will ensure that Kafka is started successfully as shown below in the image.
• The default port Number of Kafka Broker (Server) is 9092  as shown below in image.

3. Create a Topic

• Open the another terminal and must ensure that you are present in same directory where the un-tar Kafka file is present.
• The following command is used to create Topic.
• (bin/kafka-topics.sh –zookeeper {zookeeper_ip}:2181–create –topic topicName  –partitions 1 –replication-factor 1  ). 
• The command is executed over shell script file (kafka- topics.sh) which takes five parameters as an argument .
• First parameter is Zookeeper address i.e. IP and Port Number.  If You are running Kafka on local machine then the IP would be localhost and Port Number will be default i.e. 2181
• The second parameter is create command  which will be used to create the topic.
• The third parameter is keyword topic along with Topic Name. You can take any Name as a Topic Name. I have taken the topic Name as KafkaLearning as shown below.
• Next Parameter is Number of partitions you want to create for your topic. As per your requirement you can create any number of partitions.
• Last parameter is replication factor. Replication Factor means the number of copies you want to create for your partition. I will discuss it in detail in next part.
• After writing the command, Press Enter.
• If everything would be fine then you will get the confirmation message as shown below in image.

4. Start a Console Producer

• Open the another terminal and must ensure that you are present in same directory where the un-tar Kafka file is present.
• The following command is used to start a Console Producer who will be sending messages to the Kafka.
• bin/kafka-console-producer.sh –broker-list {kafka_ip}:9092 –topic topicName
• The command is executed over shell script file (kafka- console-producer.sh) which takes two parameters as an argument .
• First parameter is Kafka Broker address which contains the IP and Port Number of Kafka Broker. you have to use keyword broker-list along with IP or localhost (If you are running on Local Machine) and default Port Number i.e. 9092.
• The second parameter is topic Name which will be storing all the messages sent by Producer. The keyword topic is used along with topic Name  as shown below in image.
• Press Enter after writing the command on terminal and you will not get any confirmation or any message. The cursor will start blinking over the terminal as you press enter and that will ensure you the producer is started successfully.
• Note: Please for the Time being ignore the following messages shown below in the image. In the last step you will automatically understand what is the meaning of these messages.

5. Start a Console Consumer

• Open the another terminal and must ensure that you are present in same directory where the un-tar Kafka file is present.
• The following command is used to start a console Consumer who will be receiving  the messages from Kafka.
• bin/kafka-console-consumer.sh –bootstrap-server {kafka_ip}:9092 — topic topicName
• The command is executed over shell script file (kafka- console-consumer.sh) which takes two parameters as an argument .
• First parameter is Kafka Broker address which contains the IP and Port Number of Kafka Broker. you have to use keyword bootstrap-server along with IP or localhost (If you are running on Local Machine) and default Port Number i.e. 9092.
• The second parameter is topic Name which will be storing all the messages sent by Producer. The keyword topic is used along with topic Name  as shown below in image.
• Press Enter after writing the command on terminal and you will not get any confirmation or any message. The cursor will start blinking over the terminal as you press enter and that will ensure you the consumer is started successfully.
• Note: Please for the Time being ignore the following messages shown below in the image. In the last step you will automatically understand what is the meaning of these messages.
• Note: There is another argument you can pass over the command is from-beginning. lets suppose you have not started consumer and you have only producer with you and you are keep on sending messages to consumer. Now you want to see all the message that have been send earlier when consumer was close. So you will use from-beginning keyword along with the command and it will display all the earlier messages over the consumer terminal.
• Note: bin/kafka-console-consumer.sh –bootstrap-server {kafka_ip}:9092 — topic topicName — from beginning

6. Send and Receive Messages

• Now you are ready with console producer and consumer. Its time to send and receive messages.

• See the images above for both producer and consumer. you will see some messages over both the images. Yeah, those are messages which you will be sending through producer terminal and as a response you will be getting the same messages over consumer terminal

Hopefully, now you have an idea of how to implement Kafka over Linux terminal.  I will discuss some more beautiful concepts of Kafka in upcoming parts. Till then Keep reading Kafka

 

Comments

comments