Because there is a system in which the newspaper middleware Kafuka is used in the product, the understanding and the problem encountered in the work are summarized. It is convenient for later viewing, good memory is not as bad. Kafuka is a distributed, partitioned, copied submissive release subscription message system, using Kafuka to make some simple understanding of some of them.
First, Kafuka foundation
1, Topic Topic: Kafka is used to distinguish between different category information. Speed ​​by producer
2, Producter: Publish the message to Kafka-specific Topic objects
3, consumers: Objects to subscribe and handle messages in specific Topic
4, Broker (Kafka Service Cluster): The published message is saved in a set of servers, called Kafka clusters. Every server in the cluster is a proxy. Consumers can subscribe to one or more topics and draw data from Broker, thus consumed these published messages. It is popular that Broker is a server, a node.
5, Message: Message, is the basic unit of communication, each product can send some messages to a Topic (topic).
6, Partition: Topic physically packet, a Topic can be divided into multiple partition, each partition is an ordered queue. Each message in partition will be assigned an orderly ID (Offset)

The partition is specifically in the server. At first is a directory, one of the topics There are multiple partitions below, and these partitions are stored on different servers, or in fact, there is a different directory on different hosts. Multiple partitions multiple threads, multiple thread parallel processing improve performance, where Topic topics are only logical concepts, while Partition is a distributed storage unit, which is the basis for massive data processing.

Second, Kafuka cluster architecture
Create a topic of Topica, 3 partitions are stored in different servers, which is Broker. Topic is a logical concept that cannot be drawn directly in the picture.

Note: Kafka has no copy mechanism before version 0.8, so you will lose data when you face server downtime, so try to avoid using this version Kafka.

Consumer Group - Consumer Group
We specify a Group.ID in the code when consumer data. This ID represents the name of the consumer group, and this group.id is not set, the system will set up: conf.SetProperty ("group.id" , "Chenqi");

consumerA: group.id = a
consumerB: group.id = a
consumerC: group.id = b
consumerD: group.id = b
For the consumer group, if consumera is now spending data in Topica, let consumerb also spend Topica's data, it is not consumed, but we re-specify another group.id in consumerc, consumeerc is Consumption to Topica data. And consumerd is not consumed, so in Kafka, different groups can have the only consumer to consume the same topic. So the consumer group is to exist in parallel consumption information in parallel, and they will not consume the same message.

 Consumers will connect directly to Leader, so they consume three Leaders, so a partition will not allow multiple consumers in the consumer group to consume, but in the case of consumers, a consumer is You can consume data for multiple partitions.

Third, combined with practical problems
The Kafuka cluster used in actual work is four partitions, a consumer group, but consumers have four hosts because the expansion of traffic needs to increase consumers to 8, but when the new host test, the old host It is possible to accept the Kafuka message, but the new host does not receive the Kafuka message, if the old host process is stopped, the new host starts from the new, the four hosts started by the kafuka, and the first startup is not To the message, combined with the above analysis, it is not difficult to see because the number of partitions is less than the number of consumers in the current consumer group. In the same consumer group, the number of consumers will have a space after the number of partitions, consumers will have time = partition Number of consumers, so there are four hosts that cannot receive messages. At this time, it is necessary to combine Kafuka's performance to increase the number of partitions. It is best to partition = consumer number, at this time, the highest efficiency.