Install Kafka and Kafka Manager using docker-compose

Kafka and Kafka Manager can be installed in different ways. Using the docker-compose way makes the installation operating system agnostic. In this post, we will install Kafka and Kafka Manager using docker-compose. Three services will be installed

• Kafka
• Zookeeper
• Kafka Manager

• Create Kafka Docker Compose YAML file
  • Zookeeper
  • Kafka
  • Kafka Manager
• Start the containers using docker-compose
• Open Kafka Manager
• References

Create Kafka Docker Compose YAML file

touch kafka-docker-compose.yaml

Put the below contents in that file and put your machine IP address in the **[machine-ip-address]**

version: "3"
services:
  zookeeper:
    image: zookeeper
    restart: always
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
    environment :
      ZOO_MY_ID: 1

  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: [machine-ip-address]
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

  kafka_manager:
    image: hlebalbau/kafka-manager:stable
    container_name: kakfa-manager
    restart: always
    ports:
      - "9000:9000"
    depends_on:
      - zookeeper
      - kafka
    environment:
      ZK_HOSTS: "zookeeper:2181"
      APPLICATION_SECRET: "random-secret"
      command: -Dpidfile.path=/dev/null

Three docker services are created using this docker-compose file

Zookeeper

• Zookeeper manages the Kafka broker and is running at port 2181

Kafka

• we are using the wurstmeister/kafka image for creating a Kafka docker container.
• Kafka is running at 9092
• Note: In the KAFKA_ADVERTISED_HOST_NAME configuration, you need to provide your machine IP Address
• Kafka will connect with Zookeeper and the destination of Zookeeper is provided in Kafka configuration using KAFKA_ZOOKEEPER_CONNECT

Kafka Manager

• Kafka Manager is the graphical user interface for Kafka and is running at 9000 port

Start the containers using docker-compose

docker-compose -f kafka-docker-compose.yaml up

This will start the container in foreground mode where all the logs will be printed on the console and if you exit the console then all the docker services will shut down.

If you want to start the services in background mode then you can use the -d option which is called daemon mode

docker-compose -f kafka-docker-compose.yaml up -d

Open Kafka Manager

• Go to your browser and open the Kafka manager using your address and 9000 port
• Add your cluster

• Add your cluster config
• give a suitable name to the cluster
• Add the zookeeper’s address as your-ip-address:2181

References

• YouTube Channel
• Kafka

Next: Data Pipeline using Kafka, Elasticsearch, Logstash, Kibana