MQTT Performance Tests: Cluster Survivability

Killing Each Node Outright, Under Load

A throughput number says nothing about what happens when a node dies. This run asks a different question: with the cluster carrying traffic, does killing a node outright stop the cluster from serving? Every node is killed with kill -9 — no handover, no lease released, nothing told to the cluster in advance — left down for 20 seconds, and started again. Then the next node, and so on.

The stand is the same four-node AWS cluster as the throughput results: onec5n.2xlarge per node in us-east-1a, the shared Redis and the accounts database outside the broker processes. Killing the broker here is killing a process, not a machine, which is what makes the question about the cluster rather than about AWS: the other three nodes, the store and the network are all still there.

The run takes 600 seconds, with one crash every 90 seconds: 6 crashes in all, each node in turn and the first two again. The two nodes carrying the load — publishers on one, subscribers on the other — are among the victims, so the test covers losing the node clients are attached to, not only a quiet one. The timeline is fixed rather than random: the same nodes are killed at the same seconds on every run, so a failure can be reproduced.

What the run does

The load is 100 publishers on one node and 100 subscribers on another, 200 topics, one subscriber each, 2 messages per second per publisher at QoS 1 — about 200 messages per second routed across the cluster, over the cluster links. While that runs, each crash cycle checks three things:

• the survivors keep serving — a client connects to each surviving node, every 5 seconds, for the whole outage;
• the survivors keep delivering — a message is published on one survivor and read on another, every 10 seconds, crossing the cluster while a node is gone;
• the victim carries traffic again — once it is back, the same connect and delivery checks run against it.

The geometry changes with each cycle, so every node in the cluster is at some point a survivor, a publisher's node, a subscriber's node, and the victim. A run also fails if a node's own log shows a crash, or if the cluster loses a member or its coordinator by the end.

Result

What this does and does not say

Publishing continued throughout. Clients whose node was killed reconnect on their own; the load's throughput dips while they do, and the delivery latency in the raw log is measured across the outage, so it is not a latency result and is not published as one. This page reports pass or fail on the cluster's behaviour, not a rate.

The chaos load uses clean sessions (clean_session=true) on both publishers and subscribers. It asks whether the cluster keeps serving while nodes die; it does not ask whether a disconnected client's session state moves to another node. A client whose node is killed comes back and subscribes again, as a clean session does. Session migration with a persistent session is measured separately on the Cluster page.

The four-node AWS throughput results are on the Cluster page. The runner, the load and the generated timelines are in the repository under LoadTest/Cluster/, and the systemd adapter used on these nodes is LoadTest/AWS/chaos-systemd-node.sh, so the run can be repeated as it was made.