Kafka with Spring Cloud Stream gives you the power of Kafka with familiarity and added abstraction of Spring framework. An additional advantage of using Spring Cloud Stream is that you can switch and plug other middle ware from Kafka to RabbitMQ or other supported implementations very easy.
To get going create a Spring boot project from Spring Initializr website and add cloud-stream and Kafka as dependencies. This will bring in the following dependencies.
|
1 2 3 |
compile 'org.springframework.cloud:spring-cloud-stream' compile 'org.springframework.cloud:spring-cloud-stream-binder-kafka' compile 'org.springframework.kafka:spring-kafka' |
As soon as Spring Cloud Stream detects above Kafka binder in its classpath, it uses it and knows Kafka is used as the middleware.
Kafka Docker Image set up
While there are some options for the image, I found Spotify Kafka image to be easy to use, primarily because it comes bundled with Zookeeper and Kafka together in a single image. Run the following command from your Docker machine to install and run.
|
1 |
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=localhost --env ADVERTISED_PORT=9092 spotify/kafka |
Note that if you are running on a remote machine, add an entry to your host file or modify it as below. ‘dmira’ is the name of the server.
|
1 |
docker run -d -p 2181:2181 -p 9092:9092 --add-host "dmira:127.0.0.1" --env ADVERTISED_HOST=dmira --env ADVERTISED_PORT=9092 --name eqrkafka spotify/kafka |
If you do not do it, you will get WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) . By adding –add-host to Docker run command, you are resolving the machine name to localhost. This is important because ADVERTISED_HOST is used as the hostname to publish to ZooKeeper for clients to use.
Now test to see if you can connect to Kafka on the Docker server from your local machine by creating a
|
1 |
kafka-topics.sh –create —dmira:2181 –replication-factor 3 –partitions 3 –topic testTopic |
You should get a confirmation message. You can verify the list of topics with
|
1 |
kafka_2.11-0.10.1.0/bin/kafka-topics.sh –list –zookeeper dmira:2181 |
In the next part, we will discuss how to set up Spring Cloud Stream with Kafka and post messages.
The following is the video version of this post.
