Does the message queue lose the message?

The answer is yes, so for business rigorous data, we must ensure that it is safe in the message queue and cannot be lost.

To solve the problem that you don't lose, you must first figure it out.How is the message lost?

There are 3 key points for the message:

• Producer process of sending messages
• Message queue message storage
• CONSUMER The process of consumer messages

Let's take a look at how to lose, and solutions.

It will be described in two commonly used messagesystems of Rabbitmq and Kafka.

1. Producter Loss Message

Producer sent a message to MQ, very simple, and it is over.

However, there is dangerous, such as network problems, etc. in the transmitting diagram, resulting in Normal receiving MQ.

How to solve it? The idea is very simple, let MQ send oneAccept confirmation statement (ACK) Just, just like the delivery needs to be signed.

E.g RabbitMQThere are two ways to ensure the security of sending messages.

1) Transaction message

Before the product is sent, turn on the transaction before sending a message, and then send it.

If RabBitmq does not receive a message normally, Producer will receive exception information and rollback transactions.

If it is received, Producer is submitted to the transaction.

Very reliable, but low efficiency, because this transaction mode is synchronized, it will cause blocking.

2) Confirm confirmation mode

The Producter turns on the Confirm mode. When sending a message, Rabbitmq assigns a unique ID to this message.

After successful writing, Rabbitmq sends an ACK message to Producer, indicating that the message of this ID has been successfully sent.

The CONFIRM mode also has a callback mechanism, and Producer can prepare a failed interface for Rabbitmq to call when receiving failed.

Producter receives failed notifications, or timeout, you can perform the corresponding processing logic, such as reissue.

The Confirm model is asynchronous, more efficient than transaction messages, more widely used.

Kafka Also useful ACK mode, easy to use, as long as configuration:

ack=all

Make sure Kafka will send a confirmation notification after fully receive success, so it will not be lost.

2. MQ is lost during storage

After MQ successfully receives the message, it needs to be saved, waiting for consumer consumption.

You can also lose your message during this storage.

This is usuallyMQ failurecaused.

RabbitMQ If you want to ensure that the message is not lost, you need to open it.PersistenceThe message will be written to disk.

Even if Rabbitmq is down, as long as the disk is fine, then you can load the message after reboot.

If you want further security messages, you need to configure the Rabbitmq mirror cluster to ensure high availability.

Kafka It is a natural distributed system, and Topic is divided into multiple partitions, and each partition has multiple copies.

Multiple copies of Partition, divided into Leader and Follower.

Leader is responsible for handling the reading and writing of the message, FOLLOWER is responsible for backup.

Front mentioned Kafka configurationack=allIt is to tell Kafka, Leader and all Follower all have received, only to send ACK confirmation, only Leader's own success is not.

Otherwise, if the Leader receives the completion tells the product OK, before the Leader is synchronized to Follower, the Leader is down, Kafka will select a new Leader from Follower. So, the old Leader did not have a synchronized message before the end of the past.

In order to protect the security of the message, these four parameters should be set:

replication.factor

Used to specify the number of partition copies, must be greater than 1, that is, there are at least 2 copies, a Leader a FOLLOWER.

min.insync.replicas

Used to specify several copies successfully write onlySubmit a messageOnly the message submitted can be consumed by consumer.

This value is at least greater than 1, so that at least one copy outside of Leader is synchronized to this message, not afraid of Leader.

acks=all

Used to specify several copies to send an ACK to the ProductER after receiving the message. For example, the value is 1, indicating that the Leader receives, "all" means "all copies", or written "-1" and is equivalent to "all".

retries=999

The number of retries used to specify the failure of Producer, which can be set to a large number, indicating that the failure is retry, and the promotion of success is changing.

3. Consumer Loss Message

For example, Consumer successfully received message "123", MQ will remove this message.

But before consumer processing this message, it is downtime.

After the consumer reboots, continue from MQ, this time it is the next message "124", then "123" is lost.

Therefore, consumer is not enough to receive the message.Successful processing completionTalent.

This is related to the MQ consumer confirmation mechanism.

RabbitMQ By default, the consumer successfully received the message, the ACK confirmed, Rabbitmq considered that the consumption was successful.

Turn off automatic consumer ACK, change to manually send a confirmation notification.

KafkaThe consumer sent is not ACK confirmation, but the offset tells Kafka which position has been consumed.

By default, the CONSUMER is automatically submitted to OFFSET, so it is also necessary to turn off and change it manually.

Tie upIf the news is not lost, you need to send a message to confirm that the transmission is successful. If the MQ is stored, if it is highly reliable, consumer consumption, it cannot be confirmed, and the true processing is complete.

Recommended reading

OAUTH2 diagram

Easily understand Kubernetes core concept

The architecture technology trend of developers must understand: Service Mesh