This article mainly introduces a high availability solution for Redis clusters through Jedis&Sentinel. This solution requires Jedis 2.2.2 and above (mandatory), Redis 2.8 and above (optional, Sentinel first appeared in Redis 2.4, Redis 2.8 Sentinel is more stable), Redis cluster is built in the form of sharding plus master-slave, to meet the requirements of scalability;

 

Introduction to Redis Sentinel

Redis Sentinel is the official cluster management tool provided by Redis. It has three main functions:
Monitoring, can continuously monitor whether the master and slave instances of Redis are working properly;
Notification, when the monitored Redis instance has a problem, can notify the system administrator or other program through the API;
Automatic failback, if the primary instance is not working properly, Sentinel will initiate a failback mechanism to promote a secondary instance to the primary instance, other secondary instances will be reconfigured to the new primary instance, and the application will get a new replacement address. announcement of.
Redis Sentinel is a distributed system that can deploy multiple Sentinel instances to monitor the same set of Redis instances. They use the Gossip protocol to determine a master instance downtime, perform the fault recovery and configuration changes through the Agreement protocol, and generally deploy in a production environment. Multiple instances to improve system availability, as long as a Sentinel instance is running properly, the monitored Redis instance will behave normally (similar to Zookeeper, with multiple Zookeepers to improve system availability);
This article does not cover the implementation details and working principles of Sentinel. Readers can read other articles to understand;

 

Redis HA solution

The key to HA is to avoid single point of failure and failure recovery. Before Redis Cluster is released, Redis is generally deployed in master/slave mode. The applications discussed here are mainly used for backup, and the main instance provides read and write. Many applications are Read and write separation, read and write operations need to take different Redis instances, the program can also be used for this application, the principle is the same, the difference is how the data operation layer is encapsulated), this way to achieve HA mainly has the following solutions :
1, keepalived: through the keepalived virtual IP, providing unified access to the master and slave, when the main problem occurs, the script will be promoted by keepalived, and the master will automatically become the master after the master recovers. The advantage of this scheme is After the master-slave switchover, the application does not need to know (because the virtual IP of the access does not change), the disadvantage is the introduction of keepalived to increase the deployment complexity;
2, zookeeper: monitor the master-slave instance through zookeeper, maintain the latest valid IP, the application obtains IP through zookeeper, and accesses Redis;
3, sentinel: Sentinel monitors the master-slave instance and automatically recovers the fault. This scheme has a defect: because the master-slave instance address (IP&PORT) is different, the application cannot know the new after the master-slave switchover occurs. Address, so new support for Sentinel is added in Jedis 2.2.2. The Jedis instance obtained by the application via redis.clients.jedis.JedisSentinelPool.getResource() will be updated to the new primary instance address in time.
The author's company first used the program 1 for a while, and found that keepalived will cause data loss in some cases, keepalived through the shell script for master-slave switching, configuration is complex, and keepalived becomes a new single point, and later chose option 3, Use Redis official solution; (Scenario 2 needs to write a lot of monitoring code, no solution 3 is simple, online use program 2 readers can view it by themselves)

Problems with Sentinel

Sentinel & Jedis seems to be the perfect solution, this sentence is only half right, this is the case without fragmentation, but our application uses data fragmentation-sharing, the data is evenly distributed to 4 different instances. Each instance is deployed in a master-slave structure. Jedis does not provide a Shardnel-based ShardedJedisPool. That is to say, in 4 shards, if one of the shards has a master-slave switch, the ShardedJedisPool used by the application cannot be notified. Fragmentation operations will fail.
This paper provides a Shardnel-based ShardedJedisPool, which can sense the master-slave switching behavior of all fragments in time, and perform connection pool reconstruction.ShardedJedisSentinelPool.java

 

ShardedJedisSentinelPool for analysis

Constructor

Similar to the previous Jedis Pool constructor, the parameter poolConfig is required to provide configurations such as maxIdle, maxTotal. The masters are a List that holds the names of all the shard Masters configured in Sentinel (note that the order of the master cannot be changed because Shard The algorithm is calculated based on the location of the slice. If the order is wrong, the data store will be confused.) The sentinels are a Set, which stores all the addresses of Sentinel (format: IP:PORT, such as 127.0.0.1:26379), regardless of the order;

Initialize the connection pool

In the constructor, by method

Get the master address (IP&PORT) of all the current shards. For each shard, connect the Sentinel instance in sequence to obtain the master address of the shard. If it is not available, all Sentinel cannot be connected. It will sleep for 1 second and continue. Try again until you get the master address of all the shards. The code block is as follows:

by

Initialize the connection pool, all connections in this connection pool point to the master of the slice;

Monitor each Sentinel

In the method

Finally, a Thread is started for each Sentinel to monitor the changes made by Sentinel:

The thread's run method subscribes to the "+switch-master" channel to the Sentinel instance via the Jedis Pub/Sub API (implementing the JedisPubSub interface and subscribing through jedis.subscribe). When Sentinel performs a master-slave switch, the thread gets the new master. The address notification determines which slice is switched by the master name, replaces the new master address with the address of the original location, and calls initPool (List masters) to rebuild the Jedis connection pool; all subsequent connections obtained through the connection pool point to the new one. Master address, transparent to the application;

Application example

 
to sum up

In this paper, through the problems encountered in the real world, that is, in the case of Redis data sharding, how to make the master-slave switch transparent to the application when using Sentinel to do HA, and to monitor multiple times simultaneously through Jedis's Pub/Sub function. The master-slave switchover of the shards reconstructs the connection pool by listening to the new address, and all subsequent connections taken from the connection pool point to the new address. The key to the solution is to use sentinel for HA. The Jedis version must be 2.2.2 and above. All connections to the Redis instance must be obtained from the connection pool.

The GitHub home page for the project:https://github.com/warmbreeze/sharded-jedis-sentinel-pool