Message Persistence: Durable and Super-Durable

A persistent MQTT broker promises that a QoS 1 or 2 message for a persistent session is not lost. How far that promise reaches depends on what fails, and it is kept in two places: the broker, which writes a record of each delivery to Redis, and Redis, which decides when those records reach the disk. Each has its own setting, and together they make two modes worth naming.

This page describes XMQ 0.9.19 and later. The Redis half applies to any version. The broker half does not: before 0.9.19, persistent messages were handed to a delivery queue after the publisher had been acknowledged, and a crash could lose that queue as well as the max_queued_writes window.

What each mode can lose

The last column is what an unconfigured Redis does: it saves a snapshot now and then, and whatever happened since is gone if the machine goes down. No broker setting can make up for that, which is why Redis's own configuration is half of this page.

The broker's window: max_queued_writes

persistence.max_queued_writes is how many delivery records may be waiting for Redis at once. While the window has room, messages are delivered without waiting for their records, so writes to Redis overlap instead of queueing one behind another, and a record whose message the subscriber acknowledges before the record is due to be written is never written at all. When the window is full, the broker stops reading from the publishers that would add to it until Redis catches up, so a slow Redis slows the publishers down and nobody else.

The same number is the most the broker can lose if its process dies: records it had not yet handed to Redis. At 0 every message waits for its own record, and nothing is lost with the broker - at the price of one Redis round trip per message, which caps the rate well below what the other settings allow. 1000 is the value both modes below use: at 60,000 messages a second it is under 20 milliseconds of traffic.

It is set on the Persistence screen of the configuration interface, or in xmq_server.conf:

"persistence": {
    "enabled": true,
    "redis_uri": "redis://localhost:6379",
    "max_queued_writes": 1000
}

Durable

Redis appends every write to its log and has the operating system flush the log to disk once a second. Losing the machine loses at most that second; losing only the Redis process loses nothing, because what it wrote is already with the operating system. This is the mode for almost every deployment that needs persistence at all: it costs little over not syncing, and the gap it leaves is one second on a power failure.

# /etc/redis/redis.conf
appendonly yes
appendfsync everysec

Super-Durable

Redis flushes its log to disk before it confirms a write. Nothing Redis confirmed is lost, whatever fails, and what the broker can lose is exactly its window. The price is a disk flush in the path of every batch of writes, so both the rate a broker can sustain and its latency depend on how fast the disk completes a flush. On a disk without power-loss protection a flush waits for the data to reach the medium; on an enterprise SSD with power-loss protection it returns as soon as the drive has the data, and is many times faster. Choose this mode where losing a single acknowledged message costs more than the throughput - and run it on such a disk.

# /etc/redis/redis.conf
appendonly yes
appendfsync always

After changing Redis

Restart Redis for the configuration to take effect (sudo systemctl restart redis-server on most Linux distributions). The first start with appendonly yes creates the log from the data Redis already has. The Docker composition in the XMQ repository (docker/docker-compose.yml) starts Redis with appendonly yes, and so runs Durable out of the box.

What each mode costs in throughput and latency, measured, is on the Persistence test page.