SpringBoot-Kafka Tutorial
Apache Kafka Broker Introduction
Kafka broker is nothing but kafka server that runs Kafka processes, and kafka broker is a program that you run either on your computer or any other computer that you want to work as Kafka server. If needed, you can start up a new server in Amazon Cloud or in Google Cloud, for example, or in some other cloud environment, and then run Kafka Broker software application on that computer in the cloud. So Kafka Broker can be a physical computer, or it can be a virtual machine that runs Kafka processes, to make your system always available and fault tolerant, you can start multiple brokers in a cluster. This way, if one of your Kafka brokers is down, you still have other brokers running and servicing requests from microservices.
Behind the scenes, these brokers work together to make sure that your events are stored reliably. There will be one broker that will act as a leader, and all other brokers will act as followers. Each of these brokers hosts Kafka topics, and each Kafka topic stores events in partitions. The leader will handle all read and write requests for the partitions, while followers will passively replicate leader's data. This way, Kafka can achieve high availability and fault tolerance.
Now, the leader and the follower roles are not fixed to a single broker, but can change dynamically depending on the availability of the broker If a leader goes down for some reason, one of the follower brokers will become a new leader and will continue servicing requests. So Kafka Broker is a software application that you will run on one or more computers. Kafka broker will handle all the work to accept messages from producer, store it reliably in topic partitions, and replicate it across other brokers in the cluster. It will also handle all the work to enable consumer microservices read events from topic partitions. Kafka broker is customizable and you can configure its behavior using configuration file.
Leaders and Followers roles
As per above discussion, I've mentioned that in Kafka cluster one, Kafka Broker will act as a leader and other Kafka brokers will act as followers. In the below diagram, I have three Kafka brokers. One Kafka broker acts as a leader, and other two Kafka brokers act as followers.
What is Leader?
A leader is responsible for handling all read and write requests for partitions in a topic. It's called a leader because it leads or it manages all operations for topic partitions. If your Kafka client wants to write a message into a topic partition, it will need to go through a leader. And once the message is successfully persisted in topic partition, it will be replicated to other Kafka brokers that act as followers. This replication process is controlled and consistent.
Followers replicate data from the leader in exact order it was written, maintaining data consistency across cluster. So leader is a single source of truth for all writes and reads in topic partition. If followers were allowed to accept writes, it could lead to inconsistencies and conflicts because different followers might have different versions of the data.
What is Follower?
A follower in Kafka is a replica of a partition. It copies all the data from the leader. The main reason it is called a follower is because it follows a leader, meaning that it replicates whatever data the leader has. The followers are very important for redundancy and fault tolerance. If leader fails, one of the followers can take over. But if you have only one broker in the cluster and if that broker goes down, then your entire system is down and there is no replicas of your data.
So having multiple brokers in the cluster allows you to have more followers, and this helps you to ensure high availability of the data. If one broker including leader goes down, then data is still available through other brokers. Followers also allow you to scale your Kafka cluster horizontally. As you add more brokers, you can increase the number of followers and thereby enhance data redundancy and the capacity to handle more data and more clients. But it is not always that the same Kafka broker is a leader, and all other Kafka brokers are always followers. If the same Kafka broker is always a leader, this will mean that all read and write operations are always done through the same server, and this would create a bottleneck.
How to avoid bottleneck
Every Kafka broker can be a leader and a follower at the same time. Let's take a look in the below diagram, on the right side in below diagram we have three kafka brokers Each broker is labeled with node ID and on the left side in the diagram I have two kafka topics that are stored on these brokers.
Now a very important detail in Apache Kafka is each partition will have a leader assigned to it, and a leader to a partition is assigned when the topic is created. When you create a topic in Kafka, it also involves specifying the number of partitions for that topic. Kafka, through its internal processes, assigns a leader for each partition right away, so each partition will have its own leader and followers.
For example, in below diagram, partition zero in topic one is assigned to work with Kafka Broker one as a leader. Partition one in the same topic is assigned to work with a different broker as a leader, and partition two in the same topic is assigned to work with a different broker as its leader, and similarly with another topic.
Topic two, in below diagram, each partition in topic two will have different broker assigned to it as a leader, and this way there is no bottleneck. There is no single Kafka server that always acts as a leader, and if one Kafka server goes down, Kafka will then reassign leaders and followers to make sure that each partition has a healthy Kafka broker to work with as a new leader.
