Basic Kafka Producer, How to write records in Java
You will learn
- How to configure Kafka Consumer properties
- How to create Kafka consumer
- How to send records with Kafka Consumer
The first example shows how to print out records from Kafka to the console.
We will have to set the properties for a Kafka Consumer Object and create it.
Then we subscribe to the topics of our choice
Setting Properties
Properties props = new Properties();
props.put(“bootstrap.servers”, “localhost:9092”);
props.put(“acks”, “all”);
props.put(“retries”, 0);
props.put(“batch.size”, 16384);
props.put(“linger.ms”, 1);
props.put(“buffer.memory”, 33554432);
props.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”);
props.put(“value.serializer”, “org.apache.kafka.common.serialization.StringSerializer”);
|
Create an Instance of a Kafka Producer, with the properties we just set
Producer producer = new KafkaProducer<>(props);
|
Cal the send() function on the producer object to send a record to the Kafka topic wetter
for (int i = 0; i < 100; i++) {
producer.send(new ProducerRecord(“wetter”, Integer.toString(x), Integer.toString(i)));
}
|
Basic Kafka Consumer example in java
In this tutorial you will learn, how to make a simple kafka consumer, which sends records for ever.
You will learn the following :
- How to create topics
- How to create Kafka Consumer
- How to configure Kafka Properties
- How to subscribe to a topic
- How to poll infinitely with a consumer
You will learn how to create topics, how to create a Kafka consumer and how to configure it.
Also, you w
The first example shows how to print out records from Kafka to the console.
We will have to set the properties for a Kafka Consumer Object and create it.
Then we subscribe to the topics of our choice
Setting Properties
Java
Properties props = new Properties();
props.put(“bootstrap.servers”, “localhost:9092”);
props.put(“group.id”, “test”);
props.put(“enable.auto.commit”, “true”);
props.put(“auto.commit.interval.ms”, “1000”);
props.put(“session.timeout.ms”, “30000”);
props.put(“key.deserializer”, “org.apache.kafka.common.serialization.StringDeserializer”);
props.put(“value.deserializer”, “org.apache.kafka.common.serialization.StringDeserializer”);
|
Create an Instance of a KafkaConsumer Object and call the subscribe() method to subscribe to some topics
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList(“wetter”, “bar”));
|
Ask the consumer for new records continuously
while (true) {
ConsumerRecords records = consumer.poll(100);
for (ConsumerRecord record : records) {
System.out.printf(“offset = %d, key = %s, value = %s”, record.offset(), record.key(), record.value());
}
}
|
Expert knowledge in Big Data, Data Science, Devops, tutorials and more!