Docker & Kubernetes | Seenu's AI-licious blog

Kafka with Spring Cloud Stream on Docker – part 2

This is a continuation of part 1 on Kafka, Spring Cloud Stream on Docker. In this post, we will be talking about setting up a Spring boot project and configuring binder for Kafka and produce messages. Before we delve deeper, it is important to understand few concepts in Spring Cloud Stream and how it works.

SOURCE
When a microservice is ready to publish a message, it will publish the message using a source. Source identifies the contract between the message producer and the message destination.

CHANNEL
A channel is an abstraction over the actual queue or topic. A channel name is always associated with a target queue name in configuration, which makes it easy to switch queues by changing configuration.

BINDER
Its Spring’s code that talks to a specific message platform, like RabbitMQ, Kafka etc.

To get started, add @EnableBinding annotation to the bootstrap class of your Spring boot project, you created in part 1. This will turn Spring boot project into a Spring Cloud Stream project.

@EnableBinding(Array(classOf[Source]))
@SpringBootApplication
class KafkaDemoApplication{
    def main(args: Array[String]): Unit = {
        SpringApplication.run(classOf[KafkaDemoApplication], args:_*)    
   }
}

@EnableBinding(Array(classOf[Source]))

@SpringBootApplication

class KafkaDemoApplication{

def main(args: Array[String]): Unit = {

SpringApplication.run(classOf[KafkaDemoApplication], args:_*)

}

Next, configure your application.yml as below.

spring:
  cloud:
    stream:
      bindings:
        output: #maps to source.output in DemoProducer
          destination:  kafkaDemoTopic #kafka topic
          content-type: application/json
      kafka:
        binder:
          zkNodes: dmira #kafka
          brokers: dmira #zookeeper

spring:

cloud:

stream:

bindings:

output: #maps to source.output in DemoProducer

destination: kafkaDemoTopic #kafka topic

content-type: application/json

kafka:

binder:

zkNodes: dmira #kafka

brokers: dmira #zookeeper

The above binding maps to a channel called output with Kafka topic called KafkaDemoTopic. It specifies Kafka is the underlying message platform. You can switch this to RabbitMQ, ActiveMQ etc depending on your actual implementation. It also specifies where Kafka and Zookeeper are running. Couple of things to watch out for.

Failed to create producer binding; retrying in 30 seconds.

1	Failed to create producer binding; retrying in 30 seconds.

You may get above error if port specifications are not correct. I didn’t specify ports as spring already knows default ports for Zookeeper and Kafka.

 Magic v1 does not support record headers

1	Magic v1 does not support record headers

This is a common error, if you forget to add message converters.

Once, you have the above set up, create a message producer to post the messages to the topic as below.

@Component
class DemoKafkaProducer {
    @Autowired private var source:  Source = _
    def postMessagesToTopic( )={
        val lisyStr = List("1", "2", "3")
        lisyStr.foreach { t =>
              source.output.send(MessageBuilder.withPayload(t).build)
        }
    }
}

@Component

class DemoKafkaProducer {

@Autowired private var source: Source = _

def postMessagesToTopic( )={

val lisyStr = List("1", "2", "3")

lisyStr.foreach { t =>

source.output.send(MessageBuilder.withPayload(t).build)

}

That’s all you have to do. Check your Topic for messages.

C:\kafka_2.11-2.1.0\bin\windows>kafka-console-consumer.bat --bootstrap-server dmira:9092 --topic kafkaDemoTopic
1
2
3

C:\kafka_2.11-2.1.0\bin\windows>kafka-console-consumer.bat --bootstrap-server dmira:9092 --topic kafkaDemoTopic

Please watch the video below for additional details.

Download working source code from kafka–springboot folder.

Kafka with Spring Cloud Stream on Docker – part 1

Kafka with Spring Cloud Stream gives you the power of Kafka with familiarity and added abstraction of Spring framework. An additional advantage of using Spring Cloud Stream is that you can switch and plug other middle ware from Kafka to RabbitMQ or other supported implementations very easy.

To get going create a Spring boot project from Spring Initializr website and add cloud-stream and Kafka as dependencies. This will bring in the following dependencies.

compile 'org.springframework.cloud:spring-cloud-stream'
compile 'org.springframework.cloud:spring-cloud-stream-binder-kafka'
compile 'org.springframework.kafka:spring-kafka'

compile 'org.springframework.cloud:spring-cloud-stream'

compile 'org.springframework.cloud:spring-cloud-stream-binder-kafka'

compile 'org.springframework.kafka:spring-kafka'

As soon as Spring Cloud Stream detects above Kafka binder in its classpath, it uses it and knows Kafka is used as the middleware.

Kafka Docker Image set up

While there are some options for the image, I found Spotify Kafka image to be easy to use, primarily because it comes bundled with Zookeeper and Kafka together in a single image. Run the following command from your Docker machine to install and run.

docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=localhost --env ADVERTISED_PORT=9092 spotify/kafka

1	docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=localhost --env ADVERTISED_PORT=9092 spotify/kafka

Note that if you are running on a remote machine, add an entry to your host file or modify it as below. ‘dmira’ is the name of the server.

docker run -d -p 2181:2181 -p 9092:9092 --add-host "dmira:127.0.0.1" --env ADVERTISED_HOST=dmira --env ADVERTISED_PORT=9092 --name eqrkafka spotify/kafka

1	docker run -d -p 2181:2181 -p 9092:9092 --add-host "dmira:127.0.0.1" --env ADVERTISED_HOST=dmira --env ADVERTISED_PORT=9092 --name eqrkafka spotify/kafka

If you do not do it, you will get WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) . By adding –add-host to Docker run command, you are resolving the machine name to localhost. This is important because ADVERTISED_HOST is used as the hostname to publish to ZooKeeper for clients to use.

Now test to see if you can connect to Kafka on the Docker server from your local machine by creating a testTopic.

kafka-topics.sh –create —dmira:2181 –replication-factor 3 –partitions 3 –topic testTopic

1	kafka-topics.sh –create —dmira:2181 –replication-factor 3 –partitions 3 –topic testTopic

You should get a confirmation message. You can verify the list of topics with

kafka_2.11-0.10.1.0/bin/kafka-topics.sh –list –zookeeper dmira:2181

1	kafka_2.11-0.10.1.0/bin/kafka-topics.sh –list –zookeeper dmira:2181

In the next part, we will discuss how to set up Spring Cloud Stream with Kafka and post messages.

The following is the video version of this post.

How to deploy Docker container to AWS ECS

This post complements my video on How to deploy Docker container to AWS ECS. If you haven’t seen my post on Dockerizing Microservices, I would highly recommend reading it first. This post along with the video covers the following topics.

Docker setup
Creating IAM user for ECR
ECS Cluster
Elastic Container Registry
Task Definition
Service

First log into AWS console, click on ECS (Elastic Container Service) and click on Repository to create a Docker repository. Once, the repo is created click on View Push Commands on the top right, you should see a list of commands as below.

If you have not created an IAM user, watch the video to create an IAM user with proper permissions. Type ‘aws configure’ on your terminal and log in using access key and the secret access key as below.

Then type in the following commands, as listed. These commands will be different and specific to your image and repository.

This completes and pushing your image to ECS. Please proceed to the video to complete the rest.

Dockerizing Microservices

The main advantages of containerizing the microservices are

It makes applications portable, as dependencies can be packaged together with applications. For example, if one application needs JRE7 and the other needs JRE8, they can be packaged separately with their respective dependencies and deployed on a server, regardless of what JRE version exists on the server.
It removes the bottleneck of being limited by a number of servers during the release process, as you can deploy multiple versions of the same image or multiple features at the same time. For example, you can deploy multiple features for QA at the same time, regardless of how many servers you have.
You can start multiple instances of the application to handle increasing load.
As the containers can be isolated from each other, it will help in security.

Docker is a command line program, a background daemon. Docker containers run natively on Linux and share the kernel, making it very light weight as compared to VMs.

In this post, we will containerize a Spring Microservice with Docker. First, install Docker and make sure it is working as listed at Install Docker.

Then go to Spring Initializer site and add Web as a dependency and click on Generate Project to download a template project. Import the project into your favorite IDE and add the @RestController annotation to the main class. Add additional method with @RequestMapping as shown below.

Run the application and go to http://localhost:8080 and you should see “My First Dockerized Microservice” in your browser. This ensures, our service builds and runs fine without Docker. Stop the application and close your IDE.

Add a file called Dockerfile to the project directory and copy and paste the following contents into it.

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ADD build/libs/microservice-docker-0.0.1-SNAPSHOT.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","/app.jar"]

FROM openjdk:8-jdk-alpine

VOLUME /tmp

ADD build/libs/microservice-docker-0.0.1-SNAPSHOT.jar app.jar

EXPOSE 8080

ENTRYPOINT ["java","-jar","/app.jar"]

Change the name of the jar file to your jar file in the 3^rd line. In my case it was ‘microservice-docker-0.0.1-SNAPSHOT.jar’ and leave the rest as is.

FROM tells Docker to install the image from DockerHub if it is not present locally in Docker.

VOLUME specifies a directory outside the container, where Spring Boot creates working directories for Tomcat

ADD tells Docker to add the specified jar file to the image as app.jar

EXPOSE tells Docker to expose this port to the outside world.

ENTRYPOINT tells Docker to execute the app.jar file.

That is all you need.

Now go to the Terminal and cd to the project directory and execute the following commands.

docker build -t containerizedms . (this will build Docker image, change the last argument name as you like)
docker run -d -p 8080:8080 containerizedms (this will create a container from above image and map port 8080 inside the container to 8080 of the host machine)
docker ps (check if the container is running)

docker build -t containerizedms . (this will build Docker image, change the last argument name as you like)

docker run -d -p 8080:8080 containerizedms (this will create a container from above image and map port 8080 inside the container to 8080 of the host machine)

docker ps (check if the container is running)

Click on http://localhost:8080 and you should see “My First Dockerized Microservice” in your browse.

In the 2^nd command -d tells Docker to run it as detached so your Terminal is not tied up. Port number on the left is host machine port, it can be any valid port. Docker container is an instance of an image. Think of an image as a Java class and a container as an instance of that class.

Once, you are done stop and remove containers with the following commands. Note that, each time you use Docker run, it creates a new container from the image.

docker ps (shows all the running containers along with container ids)
docker rm &lt;container id> (This will delete the container)

1 2	docker ps (shows all the running containers along with container ids) docker rm <container id> (This will delete the container)

Download Source code from Github.