Redis p99 goes from 0.4ms to 900ms. Every operation is affected, including plain GET calls against small string values. CPU on the Redis host is at 30 percent, memory is fine, the network is not saturated, and INFO shows no evictions.

Then it clears, and forty seconds later it happens again.

Somebody added an admin endpoint that lists cache keys matching a pattern. It runs every minute. It calls KEYS.

One thread, one command at a time

Redis executes commands on a single thread. That is a design decision rather than an oversight, and it is why every Redis operation is atomic without any locking: nothing else can be running while your command runs.

The consequence is head-of-line blocking with no escape. A command that takes 200 milliseconds does not slow down your request. It makes every other client wait 200 milliseconds, including clients whose request would have taken 30 microseconds.

Client B asked for one small string. It waited 340 milliseconds because of a command it has no relationship to. From B’s perspective Redis was slow, and every metric B has will say so.

Redis 6 added threaded I/O, and it is worth knowing exactly what that changed. Reading from and writing to sockets can now happen on multiple threads, which helps when a large number of connections saturates the network handling. Command execution is still single threaded, deliberately, because that is what keeps operations atomic. Threaded I/O does nothing for this problem.

The commands that are linear and do not look it

KEYS is the famous one and it is not the only one. Anything whose cost scales with the size of the data it touches will block for proportionally long.

The DEL row is the one that catches people, because deletion intuitively feels cheap. Removing a set with two million members means freeing two million objects, and that happens synchronously on the same thread everything else is waiting on. UNLINK exists precisely for this: it removes the key from the keyspace immediately and reclaims the memory in a background thread, so the blocking portion is constant.

LRANGE key 0 -1 deserves a mention because it appears in code that looks careful. Somebody wrote a helper that fetches a whole list, the list had 40 elements in development, and it now has 900,000 in production.

Finding the offender

SLOWLOG is where to look and it is the first thing I check when Redis latency moves without a corresponding change in traffic.

redis-cli SLOWLOG GET 10

It records commands exceeding slowlog-log-slower-than, which defaults to 10,000 microseconds. On a server where normal commands take tens of microseconds, 10ms is an enormous threshold and will only catch the extremes. Lowering it to 1,000 makes the log genuinely useful.

LATENCY DOCTOR and LATENCY HISTORY cover a broader set of causes, including fork pauses during persistence, which is the other common source of multi-hundred-millisecond stalls that look like nothing in the application logs.

The signature worth memorising: every operation slows at once, CPU is not saturated, and the slowdown clears without intervention. That combination almost always means one command is holding the thread. If instead one particular key or operation is slow and others are fine, the problem is elsewhere.

SCAN and the cursor contract

SCAN replaces KEYS and the difference is that it returns after a bounded amount of work.

def scan_keys(redis, pattern, count=500):
    cursor = 0
    while True:
        cursor, batch = redis.scan(cursor=cursor, match=pattern, count=count)
        for key in batch:
            yield key
        if cursor == 0:
            break

Each call examines roughly count slots and yields, so other clients get served between iterations. The whole scan takes longer in wall clock terms and blocks nobody.

Two properties of the guarantee are worth knowing before relying on it. Keys present for the entire scan are returned at least once, so you may see duplicates and must handle them. Keys added or removed during the scan may or may not appear, so SCAN gives you a fuzzy snapshot rather than an exact one. For finding keys to expire or clean up that is fine. For anything requiring exactness it is the wrong tool, and the right answer is usually to maintain an explicit index in a set rather than scanning the keyspace at all.

The same cursor pattern exists for collections: HSCAN, SSCAN and ZSCAN do for large hashes, sets and sorted sets what SCAN does for the keyspace.

Big keys are a latency problem before they are a memory problem

A single key holding a large collection is a latency risk on every operation that touches it, including its eventual deletion and its serialisation during persistence.

redis-cli --bigkeys
redis-cli --memkeys

--bigkeys samples the keyspace and reports the largest key per type. Running it during a quiet period is one of the higher value ten second investigations available, because a single 4 million element set is a stall waiting for whichever unlucky request touches it.

The structural fix is sharding the key. A set of 4 million members becomes 64 sets of 60,000 members addressed by a hash of the member, and every operation touches one shard. That also spreads the key across a cluster rather than pinning it to one node, which no amount of ring based key distribution will do for you, because the hash of a given key is fixed.

What I would put in place

Set slowlog-log-slower-than to 1,000 microseconds so the log records things that matter on a server where normal is measured in tens of microseconds.

Disable the dangerous commands outright in production rather than relying on nobody calling them:

rename-command KEYS ""
rename-command FLUSHALL ""
rename-command FLUSHDB ""

This is the rare case where I favour removing a capability over documenting it, because the cost of one accidental KEYS against a large keyspace is an outage and the benefit of having it available in production is close to zero.

Run --bigkeys on a schedule and alert when the largest key crosses whatever threshold your latency budget tolerates.

And treat the documented complexity of every command as a latency budget rather than trivia. On a single threaded server, O(N) is not a note about that command. It is a statement about what every other client will experience while it runs.