This chapter of the Redis database server implementations are introduced, indicating achieve Redis database-related operations, including key-value pairs in the database to add, delete, view, update and other operations to realize; to achieve the client to switch databases; key timeout relevant functions , space key event notifications.

         These features, space key event notifications are implemented in src / notify.c, the other functions are implemented in src / db.c in.

 

         redisServer redis.h in the data structure definition, the definition of all attributes associated redis server, which contains the structure of the database:

struct redisServer {
    ...
    redisDb *db;
    ...
    int dbnum;     
    ...
}

         Wherein, db is a structure type redisDb array of a total dbnum redisDb structures, each structure can represent a database redisDb of Redis. Dbnum The default value is 16, the value may be configured databases configuration options in a configuration file.

 

One: The client database switch

         By default, the client database using 0, the client can use the select command, which switches to use the database. redisClient redis.h structure defined in redis on behalf of the client, which is defined below

typedef struct redisClient {
    ...
    redisDb *db;
    ...
} redisClient;

Wherein the db, it means that the client database currently in use, the pointer points to an element in the array redisServer in db. Thus, implementation of the database using a select switch is very simple, just change the redisClient db points to another array member can, as follows:

int selectDb(redisClient *c, int id) {
    if (id < 0 || id >= server.dbnum)
        return REDIS_ERR;
    c->db = &server.db[id];
    return REDIS_OK;
}

         Note that so far, Redis still can not return to the command of the database client currently in use. After several switch databases, it is likely to forget that they are currently being used by which database. When this happens, the database in order to avoid erroneous operation, before performing dangerous commands such as FLUSHDB best to execute a SELECT command, explicitly switch to the specified database before executing the command.

 

Two: the database key operation

         Redis is a key to the database server, all key-value pairs are stored in the dictionary (dict) This data structure. RedisDb redis.h defined in the database, which is defined as follows:

typedef struct redisDb {
    dict *dict;                 /* The keyspace for this DB */
    dict *expires;              /* Timeout of keys with a timeout set */
    ...
    int id;                     /* Database ID */
    ...
} redisDb;

Which, dict is in the database, save all the dictionary key-value pairs; id indicates that the database index in the array db redisServer in. Expires in the dictionary, all of the stored key set timeout period, wherein the value of the timeout time of the recording key.

Note that, in the underlying implementation, the dict keys are stored in the form of the original character string, and not a string object.

 

1: Find key-value pairs

Find key-value pairs in the database are ultimately implemented by lookupKey function, its code is as follows:

robj *lookupKey(redisDb *db, robj *key) {
    dictEntry *de = dictFind(db->dict,key->ptr);
    if (de) {
        robj *val = dictGetVal(de);

        /* Update the access time for the ageing algorithm.
         * Don't do it if we have a saving child, as this will trigger
         * a copy on write madness. */
        if (server.rdb_child_pid == -1 && server.aof_child_pid == -1)
            val->lru = LRU_CLOCK();
        return val;
    } else {
        return NULL;
    }
}

         The function key corresponding to the key value of the object to find in the database the db. Find robj structure of the return value, otherwise it returns NULL.

The dictFind (db-> dict, key-> ptr), described in the database db dict, the key is stored in the form of the original string;

After finding the target value val, if there is no current or RDB AOF process, updating lru the property value of the object, that is, update the access time of the key-value pairs.

 

2: setting key pair (add or update)

Adding key database Yes, it is realized by a function dbAdd, its code is as follows:

void dbAdd(redisDb *db, robj *key, robj *val) {
    sds copy = sdsdup(key->ptr);
    int retval = dictAdd(db->dict, copy, val);

    redisAssertWithInfo(NULL,key,retval == REDIS_OK);
    if (val->type == REDIS_LIST) signalListAsReady(db, key);
    if (server.cluster_enabled) slotToKeyAdd(key);
}

First copy the original key object key string copy, then use dictAdd, add the original string copy, the value of the object val db-> dict in.

         If the value of the object is a list of objects, you can also call signalListAsReady, making calls brpop or blpop on that list key client can stop the obstruction, the corresponding values; if Redis also open the cluster function is called slotToKeyAdd save the key in number of the slot in the cluster.

         Note that if the database dictionary db-> dict already exists in the same key, and then add fail, in which case an error exit the program directly, to ensure that the work does not have the same key of the dictionary, is the responsibility of the caller; and this function keys do not increase the reference count on, which is the responsibility of the caller.

 

Updated value of a key in a database, a function is achieved by dbOverwrite, its code is as follows:

void dbOverwrite(redisDb *db, robj *key, robj *val) {
    dictEntry *de = dictFind(db->dict,key->ptr);

    redisAssertWithInfo(NULL,key,de != NULL);
    dictReplace(db->dict, key->ptr, val);
}

         This function is mainly achieved by calling dictReplace.

         Note that if the database dictionary db-> dict key is not found, the error exit the program directly, to ensure that key does exist in the dictionary of the work is the responsibility of the caller; and this function does not increase the key reference for the count, this is also done by the caller.

 

         In Redis database, setting key set based on the command, it is achieved by setKey function, its code is as follows:

void setKey(redisDb *db, robj *key, robj *val) {
    if (lookupKeyWrite(db,key) == NULL) {
        dbAdd(db,key,val);
    } else {
        dbOverwrite(db,key,val);
    }
    incrRefCount(val);
    removeExpire(db,key);
    signalModifiedKey(db,key);
}

         First call lookupKeyWrite, looking for the key in the db, find the value dbOverwrite update is called, can not find the call dbAdd Add pair;

         This function will do the following: Call incrRefCount, increase the value of the reference count object; call removeExpire, the key is removed from the db-> expires, the key to survival is clear, which is one of the effects of the set command; call signalModifiedKey for indicating that the key has been updated, so watch the key client perceived.

 

3: Delete key-value pairs

From the database to delete the key operation is achieved by function dbDelete, its code is as follows:

int dbDelete(redisDb *db, robj *key) {
    /* Deleting an entry from the expires dict will not free the sds of
     * the key, because it is shared with the main dictionary. */
    if (dictSize(db->expires) > 0) dictDelete(db->expires,key->ptr);
    if (dictDelete(db->dict,key->ptr) == DICT_OK) {
        if (server.cluster_enabled) slotToKeyDel(key);
        return 1;
    } else {
        return 0;
    }
}

First, if db-> expires is not empty, then try to remove the key from db-> expires in; then, try to remove the key from db-> dict, delete successful and the server cluster feature is turned on, then call slotToKeyDel, from jumping table server.cluster-> slots_to_keys delete records of the key.

This function deletes successful return 1, otherwise it returns 0.

 

Three: the survival time of the function keys

         By class command expire (expire, pexpire, expireat, pexpireat), the client can set the expiration time of a key database, when key expiration time comes, the server will automatically delete the key from the database.

         By ttl (or pttl) command to view the remaining lifetime of a key, that is, the key is returned from the server automatically deletes much time.

 

         The above mentioned before, the dictionary expires redisDb structure, holds all the key set timeout. Each dictionary key-value pair expires, the bond redisDb dict shared, the lifetime value of the recording of the key, are based on a survival time in milliseconds Unix timestamp is saved.

 

1: Set the timeout

In src / db.c in, setExpire function implements the key button provided the when the timeout function, as follows:

void setExpire(redisDb *db, robj *key, long long when) {
    dictEntry *kde, *de;

    /* Reuse the sds from the main dict in the expire dict */
    kde = dictFind(db->dict,key->ptr);
    redisAssertWithInfo(NULL,key,kde != NULL);
    de = dictReplaceRaw(db->expires,dictGetKey(kde));
    dictSetSignedIntegerVal(de,when);
}

         The function first looks from db-> dict in key-> ptr corresponding dictEntry structure kde, if not, directly error exit;

         Then key-> ptr is a bond to when the value, the key-value pair to the dictionary db-> expires in. When the absolute value of which represents the timeout, it is a Unix timestamp in milliseconds.

Note that the following are db- server initialization> expires dictionary code:

for (j = 0; j < server.dbnum; j++) {
    server.db[j].dict = dictCreate(&dbDictType,NULL);
    server.db[j].expires = dictCreate(&keyptrDictType,NULL);
    ...
    server.db[j].id = j;
    ...
}

Initialization dictionary db-> dictType expires when using keyptrDictType, which is defined as follows:

dictType keyptrDictType = {
    dictSdsHash,               /* hash function */
    NULL,                      /* key dup */
    NULL,                      /* val dup */
    dictSdsKeyCompare,         /* key compare */
    NULL,                      /* key destructor */
    NULL                       /* val destructor */
};

It is seen, in addition to the hash function and key matching function, other functions are set to NULL.

Therefore, in setExpire function, call dictReplaceRaw the key-> ptr added to the dictionary db-> time expires, the dictionary entry dictEntry of key members directly assigned to key-> ptr, that is to say for the same key, db-> expires and the dictEntry.key db-> dict in dictEntry.key, their values ​​are the same, all pointing to the original string.

Call dictSetSignedIntegerVal set the timeout value is the value when the direct assignment to dictEntry-> v.s64, which is stored directly in the structure of the dictEntry.

 

2: Delete key timeout

         In src / db.c in, removeExpire timeout function implements the functions of the delete key, which code is as follows:

int removeExpire(redisDb *db, robj *key) {
    /* An expire may only be removed if there is a corresponding entry in the
     * main dict. Otherwise, the key will never be freed. */
    redisAssertWithInfo(NULL,key,dictFind(db->dict,key->ptr) != NULL);
    return dictDelete(db->expires,key->ptr) == DICT_OK;
}

This function is mainly by calling dictDelete the key-> ptr delete realized from db-> expires in.

         Because, key-> ptr by db-> dict and db-> expires sharing. Initialization dictionary db-> expires in dictType is keyptrDictType, one of the "keydestructor" and "val destructor" functions are NULL, therefore, in dictDelete in, does not take key-> ptr points to release memory space, it is only the corresponding dictionary entry dictEntry, from db-> expires delete release it.

 

3: Delete key timeout

To delete an expired bond, a total of three strategies, they are:

         a, regularly delete: key expiration time set at the same time, create a timer that expires at the key time comes, perform the removal of the key immediately.

         b, inert Delete: faire key expires regardless, every time the access key from the key space are the key to check whether expired, if expired, then delete the key.

         c, regularly delete: Every so often, the program checks the database once, delete the expired key. As for how much you want to delete outdated keys, and to check how many databases, the decision by the algorithm.

 

         The timing of the policy to remove memory is the most friendly: by using a timer, the policy can guarantee expired keys will be deleted as soon as possible and release the memory occupied key expired.

         On the other hand, the timing of the policy to remove the CPU time is the most unfriendly: in the relatively large number of expired keys, it is necessary to create a large number of timers, delete the expired key this behavior may consume a considerable portion of CPU time, may cause the server response time and throughput impact.

 

         Inert policy to remove the CPU time is the most friendly: The program will check for expired keys only when the access key, which can guarantee the deletion of expired keys will only operate in the case have to do it, and delete the target current treatment is limited to the key, this strategy will not spend any time on the CPU remove other irrelevant expired key.

         On the other hand, the disadvantage inert removal policy is that it is the most unfriendly memory: If there are a lot of expired keys in the database, which expired keys and just have not been accessed, then they may never be delete, this situation is similar to a memory leak.

 

         Periodically delete strategy is a compromise between the first two policies: the policy from time to time perform a delete key operation expired, and delete operations to reduce the impact on CPU time by limiting the duration and frequency of the deletion operations performed. On the other hand, regularly delete the policy effectively reduced because of expired keys brought wasted memory.

 

         Redis server actually uses inert deletion and periodically delete two strategies, by the combined use of these two deletion policy, the server can be a good use of CPU time and at reasonable balance between avoiding a waste of memory space.

 

         Achieved by src / db.c in expireIfNeeded when inert deleted each time you visit the database of key-value pairs, will call the function to determine whether the key has expired, and if so, remove it in the expireIfNeeded If not expired, expireIfNeeded not to take any action. For example, access to key functions lookupKeyWrite to achieve a write operation:

robj *lookupKeyWrite(redisDb *db, robj *key) {
    expireIfNeeded(db,key);
    return lookupKey(db,key);
}

         Visible, before looking for the key, it will call expireIfNeeded, deletion of expired keys.

         Here is expireIfNeeded function code:

int expireIfNeeded(redisDb *db, robj *key) {
    mstime_t when = getExpire(db,key);
    mstime_t now;

    if (when < 0) return 0; /* No expire for this key */

    /* Don't expire anything while loading. It will be done later. */
    if (server.loading) return 0;

    /* If we are in the context of a Lua script, we claim that time is
     * blocked to when the Lua script started. This way a key can expire
     * only the first time it is accessed and not in the middle of the
     * script execution, making propagation to slaves / AOF consistent.
     * See issue #1525 on Github for more information. */
    now = server.lua_caller ? server.lua_time_start : mstime();

    /* If we are running in the context of a slave, return ASAP:
     * the slave key expiration is controlled by the master that will
     * send us synthesized DEL operations for expired keys.
     *
     * Still we try to return the right information to the caller,
     * that is, 0 if we think the key should be still valid, 1 if
     * we think the key is expired at this time. */
    if (server.masterhost != NULL) return now > when;

    /* Return when this key has not expired */
    if (now <= when) return 0;

    /* Delete the key */
    server.stat_expiredkeys++;
    propagateExpire(db,key);
    notifyKeyspaceEvent(REDIS_NOTIFY_EXPIRED,
        "expired",key,db->id);
    return dbDelete(db,key);
}

         GetExpire first call made from the db key timeout, the timeout if the key is not set, the direct return 0;

         If the server is currently recovering data from memory, simply return 0;

         Then in milliseconds, to obtain the current time stamp now; if this process is currently running on the slave node, then immediately return the key whether a timeout, without taking any action, because the delete action is initiated concurrently by the master node to the slave's, slave not actively delete key;

         If the key is not timed out, the direct return 0; if the key has indeed been timed out, then delete action, first server.stat_expiredkeys ++; then call propagateExpire, the timeout deletion sent to AOF documents and slave node; then call notifyKeyspaceEvent, launched expired key space notice; Finally, call dbDelete remove the key.         

 

         Regularly delete enable subsequent supplementary ......

 

4: AOF, RDB and replication process for expired keys

a: RDB function key policy processing expire as follows:

         When you create a new file RDB execute commands or BGSAVE SAVE command, the program will check the database keys, expired keys are not saved to the new RDB file;

         If the server is turned on the RDB function, then start the Redis server, it will load RDB file. If the server is running in master mode, when manned RDB file, save the file in the program will check the key, the key will be ignored expired without being loaded;

         If the server is to run from the server mode, when manned RDB file, all to save the file, whether or not expired, it will be manned to the database. However, because the main data from the server during synchronization will be cleared from the database server, so generally speaking, manned RDB expired key file will not affect the server.

 

b: AOF expired policy processing function keys are as follows:

         When the server is running in persistent mode AOF, if the database is a key has expired, but it has not been deleted or inert regularly deleted, the AOF file will not have any impact because the expired bond;

         When the key is expired or deleted periodically delete inactive, the program will append a DEL command to the AOF documents explicitly record the key has been removed. This is achieved by propagateExpire function, and the function is called in expireIfNeeded in.         

 

         AOF rewrite during the execution, the program will check the database keys, expired keys are not saved to the file after the AOF rewrite.

 

c: copy function key policy processing expire as follows:

         When the server is running under the copy mode, remove expired keys from a server controlled by the operation of the primary server:

         Master server delete an expired after the key will explicitly send a DEL command from the server to all, inform delete the expired key from the server. This is achieved by propagateExpire function, and the function is called in expireIfNeeded in.

         From the server when performing a read command sent by the client, even if the encounter expired keys will not expire delete key, but continue unexpired key image processing as key to deal expired. Only after receiving the master server sent DEL command will delete expired key from the server.

         By the main server to control uniformly key from the server by removing expired, you can ensure the consistency of master data from the server.

 

5: Implement expire class command

expire command class are ultimately implemented by expireGenericCommand function, as follows:

void expireGenericCommand(redisClient *c, long long basetime, int unit) {
    robj *key = c->argv[1], *param = c->argv[2];
    long long when; /* unix time in milliseconds when the key will expire. */

    if (getLongLongFromObjectOrReply(c, param, &when, NULL) != REDIS_OK)
        return;

    if (unit == UNIT_SECONDS) when *= 1000;
    when += basetime;

    /* No key, return zero. */
    if (lookupKeyRead(c->db,key) == NULL) {
        addReply(c,shared.czero);
        return;
    }

    /* EXPIRE with negative TTL, or EXPIREAT with a timestamp into the past
     * should never be executed as a DEL when load the AOF or in the context
     * of a slave instance.
     *
     * Instead we take the other branch of the IF statement setting an expire
     * (possibly in the past) and wait for an explicit DEL from the master. */
    if (when <= mstime() && !server.loading && !server.masterhost) {
        robj *aux;

        redisAssertWithInfo(c,key,dbDelete(c->db,key));
        server.dirty++;

        /* Replicate/AOF this as an explicit DEL. */
        aux = createStringObject("DEL",3);
        rewriteClientCommandVector(c,2,aux,key);
        decrRefCount(aux);
        signalModifiedKey(c->db,key);
        notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,"del",key,c->db->id);
        addReply(c, shared.cone);
        return;
    } else {
        setExpire(c->db,key,when);
        addReply(c,shared.cone);
        signalModifiedKey(c->db,key);
        notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,"expire",key,c->db->id);
        server.dirty++;
        return;
    }
}

         basetime represent key reference timeout time, the actual timeout obtained according to the command parameters basetime, and the client: If the function pexpireat called by expireat Alternatively, the basetime is 0, and specific timeout period provided by the client in the command parameter ; If the function call has pexpire or expire, the basetime Unix timestamp for the current provided by the client in the command parameter a relative value. Finally obtained actual timeout when; unit represents the command unit of time parameter, if UNIT_SECONDS, When it is necessary to convert milliseconds;

First, look for key in c-> db - did not directly back to the client fails information, and return;

        

         If the timeout is earlier than the current time, and the current processes running on the slave nodes, and there is no data recovery, first delete directly c-> db in the key, then server.dirty ++, then call rewriteClientCommandVector? Then call signalModifiedKey indicates that the key has been updated, so watch the key clients perceive; then call notifyKeyspaceEvent, it released "del" key space notice; after finally returned back to the client

        

If the condition is not satisfied, then the first call setExpire key disposed in c-> db timeout period; and then back to the client; signalModifiedKey then call indicates that the key is updated, so that the key of the client watch perceived; and call notifyKeyspaceEvent, it released "expire" key space notice; last server.dirty ++ and returns.

 

6: Implement ttl class command

         ttl class (ttl, Pttl) command, function ultimately achieved through ttlGenericCommand, code is as follows:

void ttlGenericCommand(redisClient *c, int output_ms) {
    long long expire, ttl = -1;

    /* If the key does not exist at all, return -2 */
    if (lookupKeyRead(c->db,c->argv[1]) == NULL) {
        addReplyLongLong(c,-2);
        return;
    }
    /* The key exists. Return -1 if it has no expire, or the actual
     * TTL value otherwise. */
    expire = getExpire(c->db,c->argv[1]);
    if (expire != -1) {
        ttl = expire-mstime();
        if (ttl < 0) ttl = 0;
    }
    if (ttl == -1) {
        addReplyLongLong(c,-1);
    } else {
        addReplyLongLong(c,output_ms ? ttl : ((ttl+500)/1000));
    }
}

First call lookupKeyRead, lookup key in c-> db, if not found, then -2 is returned directly to the client;

Then call getExpire, acquires the key timeout The expire, if the key set timeout period, the computing TTL; if the key is not set timeout, -1 directly to the client;

         If output_ms is 1, the process directly returns to the client ttl; otherwise, the need to convert ttl seconds to return to the client.

 

Four: key space realization notice

Key space notification feature, mainly notifyKeyspaceEvent function in src / notify.c of implementation. Every space is required to send notification button will call this function, the function delCommand such as deleting a key:

void delCommand(redisClient *c) {
    int deleted = 0, j;

    for (j = 1; j < c->argc; j++) {
        expireIfNeeded(c->db,c->argv[j]);
        if (dbDelete(c->db,c->argv[j])) {
            signalModifiedKey(c->db,c->argv[j]);
            notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,
                "del",c->argv[j],c->db->id);
            server.dirty++;
            deleted++;
        }
    }
    addReplyLongLong(c,deleted);
}

Visible, after each successfully removed a key, key space will be called notifyKeyspaceEvent send notifications.

         notifyKeyspaceEvent code is as follows:

void notifyKeyspaceEvent(int type, char *event, robj *key, int dbid) {
    sds chan;
    robj *chanobj, *eventobj;
    int len = -1;
    char buf[24];

    /* If notifications for this class of events are off, return ASAP. */
    if (!(server.notify_keyspace_events & type)) return;

    eventobj = createStringObject(event,strlen(event));

    /* __keyspace@<db>__:<key> <event> notifications. */
    if (server.notify_keyspace_events & REDIS_NOTIFY_KEYSPACE) {
        chan = sdsnewlen("__keyspace@",11);
        len = ll2string(buf,sizeof(buf),dbid);
        chan = sdscatlen(chan, buf, len);
        chan = sdscatlen(chan, "__:", 3);
        chan = sdscatsds(chan, key->ptr);
        chanobj = createObject(REDIS_STRING, chan);
        pubsubPublishMessage(chanobj, eventobj);
        decrRefCount(chanobj);
    }

    /* __keyevente@<db>__:<event> <key> notifications. */
    if (server.notify_keyspace_events & REDIS_NOTIFY_KEYEVENT) {
        chan = sdsnewlen("__keyevent@",11);
        if (len == -1) len = ll2string(buf,sizeof(buf),dbid);
        chan = sdscatlen(chan, buf, len);
        chan = sdscatlen(chan, "__:", 3);
        chan = sdscatsds(chan, eventobj->ptr);
        chanobj = createObject(REDIS_STRING, chan);
        pubsubPublishMessage(chanobj, key);
        decrRefCount(chanobj);
    }
    decrRefCount(eventobj);
}

         Type of the key parameters for the type of notification space, such REDIS_NOTIFY_STRING like; Event Event is a string, such as "del" and the like; key is the corresponding key, the dbid events id database.

First, by the value of server.notify_keyspace_events & type of server to determine whether to open this type of event type, which is the value of the notify-keyspace-events option is included type; event and then create a string object events based on parameters eventobj;

If the server is turned keyspace type of event notification, the type of event, the message is the event string object eventobj, channel chanobj is similar: "__ keyspace @ dbid __: key" after such a string object, and news channels determined by pubsubPublishMessage, send a message to all subscribers to channels;

If the server is turned keyevent type of event notification, the type of event, the message is key string object, channel chanobj is similar: "__ keyevent @ dbid __: event" such a string object, channels and messages determined, by pubsubPublishMessage to send a message to all the channels subscribers.

 

More redis related database implementation, can refer to:

https://github.com/gqtc/redis-3.0.5/blob/master/redis-3.0.5/src/db.c