• SpringBoot-Kafka Tutorial

Introduction Kafka CLI Kafka Producer Kafka Consumer

Apache Kafka Server Installation

Download Apache Kafka Server

To download the apache kafka server, first navigate to the kafka.apache.org and then click on the Download kafka then you will navigate the downloads page. In the download page from the latest version you will see the scala 2.13 version which is recommanded to be download, Check below image to get more understanding.


Once apache kafka server has been downloaded first unzip the folder and move the unzipped folder to some other working directory.

Start Single Apache Kafka Broker with Kraft

Before we start the kafka server we need to understand few folders and files which are available in downloaded kafka server.
kafka/config/kraft inside this folder we can see below three configuration files, which used to changes the required configurations before we start the kafka server. To start the kafka server we're using the kraft which is new way of starting the kafka server.

  1. 1. broker.properties
  2. 2. controller.properties
  3. 3. server.properties

Inside the kafka/bin folder we can see multiple .sh files those are mainly used to perform certain actions like start and stop the kafka server etc.. these .sh files are used for mac. For windows again we need navigate the kafka/bin/windows folder, inside this we can find the .bat files, use those files to perform same action.

Steps to start the kafka server

First, generate a uniqueID id which helps to generate a kafka cluster, execute below commonds step by step:

  1. 1. Navigate the kafka folder which you downloaded from above step.
  2. 2. ./bin/kafka-storage.sh random-uuid: This will generate a unique identifier that is needed to configure your Kafka cluster.
  3. 3. Next we need to format the log directories using this unique identifier. execute below command.
  4. 4. ./bin/kafka-storage.sh format -t "uniqueID" -c config/kraft/server.properties
  5. 5. Using below command we can start the kafka server.
  6. 6. ./bin/kafka-server-start.sh config/kraft/server.properties

Using above steps we can start a single kafka server, let's say if we want to start a multiple kafka servers then we need to update the server.properties file also need to create the multiple server configuration.

How to start a multiple kafka server

To start a multiple kafka servers first we need to preapre the multiple configruation files inside the kafka/config/kraft. Let's say if we want to start 3 different kafka servers then we need to create the 3 different configuration files with below name.

  1. 1. server1.properties
  2. 2. server2.properties
  3. 3. server3.properties

First navigate to the kafka/config/kraft folder you will see only server.properties configuration file. Copy this file and paste 3 times so that we can create three configuration files like above.

Inside the each configration files make below changes.

  1. Make below changes in server1.properties
    1. 1. Chanege node id like node.id=1
    2. 2. Change the listerners port to 9092 and 9093 like listeners=PLAINTEXT://:9092,CONTROLLER://:9093
    3. 3. Change the advertised listeners port to 9092 like advertised.listeners=PLAINTEXT://localhost:9092
    4. 4. Change the controller quorum voters with host and port numbers like controller.quorum.voters=1@localhost:9093,2@localhost:9095,3@localhost:9097
    5. 5. Change the log directory like log.dirs=/tmp/server-1/kraft-combined-logs
  2. Make below changes in server2.properties
    1. 1. Chanege node id like node.id=2
    2. 2. Change the listerners port to 9094 and 9095 like listeners=PLAINTEXT://:9094,CONTROLLER://:9095
    3. 3. Change the advertised listeners port to 9095 like advertised.listeners=PLAINTEXT://localhost:9094
    4. 4. Change the controller quorum voters with host and port numbers like controller.quorum.voters=1@localhost:9093,2@localhost:9095,3@localhost:9097
    5. 5. Change the log directory like log.dirs=/tmp/server-2/kraft-combined-logs
  3. Make below changes in server3.properties
    1. 1. Chanege node id like node.id=3
    2. 2. Change the listerners port to 9096 and 9097 like listeners=PLAINTEXT://:9096,CONTROLLER://:9097
    3. 3. Change the advertised listeners port to 9092 like advertised.listeners=PLAINTEXT://localhost:9096
    4. 4. Change the controller quorum voters with host and port numbers like controller.quorum.voters=1@localhost:9093,2@localhost:9095,3@localhost:9097
    5. 5. Change the log directory like log.dirs=/tmp/server-3/kraft-combined-logs

How to start multiple kafka servers

To start the multiple kafka servers please follow the steps which are guided above. First open the new terminal and navigate to the kafka/config/kraft folder and run the command to start the kafka server1. Do the same to start the kafka server2 and same do for kafka server3.

Quick steps to run the 3 kafka servers

  1. Server-1
    1. 1. Open the new terminal
    2. 2. Run following command ./bin/kafka-storage.sh random-uuid to generate the unique number. Same unique number can be used in all other places.
    3. 3. Run following command ./bin/kafka-storage.sh format -t "uniqueID" -c config/kraft/server.properties which help to format kafka server logs.
    4. 4. Run following command ./bin/kafka-server-start.sh config/kraft/server-1.properties to start the server-1
    Server-2
    1. 1. Open the new terminal
    2. 2. Run following command ./bin/kafka-storage.sh format -t "uniqueID" -c config/kraft/server.properties which help to format kafka server logs.
    3. 3. Run following command ./bin/kafka-server-start.sh config/kraft/server-2.properties to start the server-2
    Server-3
    1. 1. Open the new terminal
    2. 2. Run following command ./bin/kafka-storage.sh format -t "uniqueID" -c config/kraft/server.properties which help to format kafka server logs.
    3. 3. Run following command ./bin/kafka-server-start.sh config/kraft/server-3.properties to start the server-3

To stop the kafka server, first open the new terminal and run following command ./bin/kafka-server-stop.sh for windows ./bin/windows/kafka-server-stop.sh