The following blog will make you understand about the third approach of sending data on Kafka i.e. Asynchronous Approach. You will also see the how this approach is implemented through java programming Language.
Asynchronous Send
In this approach we send a message to kafka broker and provide callback interface to get the acknowledgement sent by the kafka. In case of success you will get RecordMetaData object but in case of failure you will get Exception. In asynchronous approach you can send as many number of messages to kafka server without getting response but keep in mind that you can control the limit on number of messages that can be send to kafka at once. The “max.in.flight.request.per.connection” controls the limit of messages that can be send at once. The asynchronous approach provides better throughput but “max.in.flight.request.per.connection” parameter limits it and the default value of the parameter is 5 and you can change it to higher as per your requirement.
1. Import Necessary java Packages.
2. Create variables which will define the topicName, key and value; It is already known to you that kafka accepts messages in the form of key and value pair. The key and value pair is nothing but its your message which you want to send to kafka.
3. Create Properties file Object and put some required configurations into it.
a) bootstrap.servers is the list of kafka brokers addresses.
b) StringSerializer is the class which is used to serialize your key and value. It is known to you that kafka accepts the data in the form of array of bytes. So the process of converting your message into bytes is known as serialization.
4) Create KafkaProducer Object
The constructor of KafkaProducer takes the properties class object as an argument.
5) Create ProducerRecord Object
The constructor of ProducerRecord takes three values as an argument. Your topicName, key and your value.
6) Now use send method of Producer and pass the argument as ProducerRecord object and the reference of class which implements Callback interface. It is important to note that the in case of success you will get RecordMetaData but in case of failure you will get Exception.
7) Create your class which implements Callback interface and overrides the onCompletion() method which takes RecordMetaData and Exception object as parameter. If you got null in exception reference then it means no exception occurs and your message successfully sent to kafka but if you get some value in exception reference then you must have get some exception and with the help of that exception you can trace the errors.
8) Lastly, run your java program and in order to test your program just run the kafka console consumer on your terminal and you will see that the message you have written in your program will be on that kafka conthe sole consumer.
If you have any confusion then please refer the whole java program as shown below.
Hope Now you get an idea to send data to kafka through Asynchronous approach. In upcoming parts i will explain about the internal working of a Kafka Producer. Till then keep reading Synchronous approach of sending data to kafka.
