The basic concepts of Kafka producers, consumers, and brokers

Kafka is a publishing and subscription-based messaging system. It is often referred to as a "distributed commit log" or "distributed streaming platform". File system or database commit logs are used to provide a persistent record of all transactions, and the state of the system can be reconstructed by rebuilding these logs. Similarly, Kafka's data is persisted and stored in a certain order and can be read on demand.

1. Kafka topology

2. Characteristics of Kafka

It also provides high throughput for distribution and subscription. It is understood that Kafka can produce about 250,000 messages per second (50MB) and process 550,000 messages per second (110MB). The number of messages mentioned here may not be particularly accurate because the size of the messages may not be consistent;

Persistence operations can be performed to persist messages to disk and store them in the form of logs, so they can be used for batch consumption, such as ETL, and real-time applications. Data loss can be prevented by persisting data to hard disk and replication.
Distributed system, easy to expand outward. All producers, brokers and consumers will have multiple, all distributed. Machines can be expanded without downtime.

The state of message processing is maintained on the consumer side, not on the server side, and can be automatically balanced in the event of a failure.
Supports online and offline scenarios.

3. Kafka's core concepts

Glossary
Producer message producer
Consumer Message consumer
ConsumerGroup consumer group, can consume messages from partitions in Topic in parallel
Broker Cache agent, one or more servers in a Kafka cluster are collectively referred to as brokers.
Topic Kafka processes different types of message sources (feeds of messages)
Partition Topic is a physical grouping. A topic can be divided into multiple partitions. Each partition is an ordered queue. Each message in a partition is assigned an ordered ID (offset).
Message is the basic unit of communication. Each producer can publish some messages to a topic.
Producers are message and data generators. The process of publishing messages to a Kafka topic is called producers.
Consumers are consumers of messages and data. The process of subscribing to a topic and processing its published data is called consumers.

3.1 The concept of Producers

Message and data generators. The process of publishing messages to a Kafka topic is called producers.
The Producer publishes the message to the specified Topic. The Producer can also decide which partition the message belongs to, for example, based on a round-robin approach or some other algorithm.
Asynchronous sending and batch sending can effectively improve sending efficiency. The asynchronous sending mode of kafka producer allows batch sending, first caching the messages in memory, and then sending them out in batches at one time.

3.2 The concept of broker:

Broker does not have a replica mechanism. Once a broker goes down, the messages of that broker will be unavailable.
Broker does not save the status of subscribers, which is saved by the subscribers themselves.
The statelessness makes it difficult to delete messages (the deleted messages may be subscribed to). Kafka uses a time-based SLA (service assurance), and messages will be deleted after being stored for a certain period of time (usually 7 days).
Consumer subscribers can rewind back to any location to re-consume. When a subscriber fails, the smallest offset (id) can be selected to re-read the consumption message.

3.3 Message Composition

Message: It is the basic unit of communication. Each producer can publish a message to a topic.
Messages in Kafka are organized based on topics. Different topics are independent of each other. Each topic can be divided into different partitions, and each partition stores a part of the message.
Each Message in a partion contains the following three attributes:
offset long
MessageSize int32
Specific content of data messages

3.4 The concept of consumers

Message and data consumers, the process of subscribing to a topic and processing the messages it publishes is called consumers. In Kafka, we can think of a group as a "subscriber". Each partition in a topic will only be consumed by a consumer in a "subscriber", but a consumer can consume messages from multiple partitions. Note: Kafka's design principle determines that for a topic, the same group cannot have more consumers than the number of partitions consuming at the same time, otherwise it will mean that some consumers cannot get the message