The following blog will let you know about some of mostly used broker configurations. Please keep in mind that broker and Kafka server are one and the same thing. Server.properties file is responsible for maintaining all the configurations related to broker.

Broker Configurations

Commonly used broker configuration parameters are :-

1. zookeeper.connect
2. delete.topics.enable
3. auto.create.topics.enable
4. default.replication.factor
5. num.partitions
6. log.retention.ms
7. log.retention.bytes

1) zookeeper.connect

The zookeeper.connect parameter takes zookeeper connection string i.e. (Address of Zookeeper). The connection string is basically a host name with port number. This parameter also plays an important role while forming a cluster. Lets assume you are forming a kafka – cluster in which you have 5-6 computer with you and on each computer an instance of kafka server is running. Now, how do all the kafka server knows about each other means how they are connected to each other. They are connected with each other with the help of zookeeper and zookeeper.connect parameter provides the link between all the instances of brokers running on different machines to form a cluster. The following image shows the zookeeper.connect value as in my case it is set to localhost:2181. You can specify the IP Address and port number on which your zookeeper is running.

2) delete.topic.enable

If you want to delete a topic in kafka then the above parameter play a key role in deleting a topic. With the help of topic management tool you can delete a topic but in kafka by default deleting a topic is not allowed because the default value of the delete.topic.enable is false. The default value of parameter is only good for production environment but for development and testing environment you can change it to true. As you change it to true then you will be able to delete a topic in kafka,

3) auto.create.topics.enable

The above parameter allows kafka to automatically create a topic. This parameter plays handy role when the producer starts sending data to kafka without creating a topic. The parameter takes two values true or false. True value will allow kafka to create topic automatically while the value false will disable the feature of automatic creation of topic. By default the value set to the parameter is true.

4) default.replication.factor & num.partitions

Both the above parameters are useful when the auto.create.topic.enable is set to be true. You have seen above that if you set the auto.create.topic.enable to true then kafka will create topic automatically. Now, if the kafka has created the topic by itself then what would be the replication factor and partitions for that topic ? There the above two parameters plays crucial role. default.replication.factor will allow kafka to set the number of replicas of that topic and by default it is set to 1. Other parameter num.partitions allow kafka to set number of partitions for that topic and by default it is also set to 1.

\

5) log.retention.ms & log.retention.bytes

The above two parameters decide that how long the data sent by producer to kafka, will remain inside the kafka. Please keep in mind that the data sent to kafka will not remain inside the kafka forever as kafka is not a database. Kafka act as a message broker as it delivers the data to consumer it should clean up the data. With the help of above two parameters you can configure the retention period. The default option is retention by time i.e. log.retention.ms and its default value is 7 days. This means that kafka will automatically clean up the data older than 7 days. You can change the value as per your requirement.

log.retention.bytes is another parameter which also does the same thing but it is related to size of data and this parameters applies on the partitions of topic. Assume that you have set the value to 1GB then kafka will start a cleanup activity which will clean the data as the partitions size reaches to 1GB. Generally, it is advised to use one of them for clean up activity but if there is no harm if you have set the values to both the parameters because then kafka will trigger the cleanup activity either meeting any of criteria.

That’s all about some of the mostly used broker configurations. How producer works internally and how we can start producer through java code will be discussed in upcoming parts. For the time being lets look on to Installation and Implementation of Kafka.

Comments

comments