Category Archives: Kafka

This contains tutorials about the messaging system Kafka, a distributed streaming plattform
These article show you :
– how to setup Kafka
– how to use basic Kafka Producer
– how to use basic Kafka Consumer
– how to use basic Kafka Broker
– how to use Kafka Topics
– how to use Kafka with Apache Spark
– how to use Kafka with Hadoop

Kafka

Basic Kafka Producer, How to write records in Java

Leave a comment

Basic Kafka Producer, How to write records in Java

You will learn

  • How to configure Kafka Consumer properties
  • How to create Kafka consumer
  • How to send records with Kafka Consumer

The first example shows how to print out records from Kafka to the console.

We will have to set the properties for a Kafka Consumer Object and create it.

Then we subscribe to the topics of our choice

Setting Properties

Properties props = new Properties();
props.put(“bootstrap.servers”, “localhost:9092”);
props.put(“acks”, “all”);
props.put(“retries”, 0);
props.put(“batch.size”, 16384);
props.put(“linger.ms”, 1);
props.put(“buffer.memory”, 33554432);
props.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”);
props.put(“value.serializer”, “org.apache.kafka.common.serialization.StringSerializer”);

Create an Instance of a Kafka Producer, with the properties we just set

Producer producer = new KafkaProducer<>(props);

Cal the send() function on the producer object to send a record to the Kafka topic wetter

for (int i = 0; i < 100; i++) { producer.send(new ProducerRecord(“wetter”, Integer.toString(x), Integer.toString(i)));

}

Kafka

Basic Kafka Consumer Java example (How to print Kafka records to terminal)

Leave a comment

Basic Kafka Consumer example in java

In this tutorial you will learn, how to make a simple kafka consumer, which sends records for ever.

You will learn the following :

  • How to create topics
  • How to create Kafka Consumer
  • How to configure Kafka Properties
  • How to subscribe to a topic
  • How to poll infinitely with a consumer

You will learn how to create topics, how to create a Kafka consumer and how to configure it.

Also, you w

The first example shows how to print out records from Kafka to the console.

We will have to set the properties for a Kafka Consumer Object and create it.

Then we subscribe to the topics of our choice

Setting Properties

Java

Properties props = new Properties();
props.put(“bootstrap.servers”, “localhost:9092”);
props.put(“group.id”, “test”);
props.put(“enable.auto.commit”, “true”);
props.put(“auto.commit.interval.ms”, “1000”);
props.put(“session.timeout.ms”, “30000”);
props.put(“key.deserializer”, “org.apache.kafka.common.serialization.StringDeserializer”);
props.put(“value.deserializer”, “org.apache.kafka.common.serialization.StringDeserializer”);

Create an Instance of a KafkaConsumer Object and call the subscribe() method to subscribe to some topics

KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList(“wetter”, “bar”));

Ask the consumer for new records continuously

while (true) {
ConsumerRecords records = consumer.poll(100);
for (ConsumerRecord record : records) {
System.out.printf(“offset = %d, key = %s, value = %s”, record.offset(), record.key(), record.value());
}
}