Credits:The XMQ web site is provided by Linotex.The XMQ and SPTK Windows installers are created with Advanced Installer free Open-Source license.
There were 0 unique visitors to this page

MQTT Performance Tests: Cluster

Four Nodes, One Client, One Shared Store

The XMQ cluster has four broker nodes, each on its own AWS c5n.2xlargeinstance (8 vCPU, 21 GiB), in one placement group in us-east-1a. A singlec5n.4xlarge client carries every connection, binding its clients to 29 source addresses. XMQ keeps its cluster state in a dedicated Redis instance and its accounts in PostgreSQL; EMQX 5.8.9 is the last Apache 2.0 release that can form a cluster without a commercial licence. Both products run MQTT 5 on port 1884, QoS 1, with a 16-byte payload.

The comparison covers only brokers that can form a cluster in their community editions. XMQ and EMQX 5.8.9 both do so without a commercial licence; brokers whose community editions do not support clustering were not tested here, because a single-node result would not answer the question this page asks.

Every scenario starts on a freshly restarted cluster, with the other broker stopped. Each node is a separate process on a separate machine, so the numbers include the cluster links, not just one broker's event loop. Latency is end-to-end delivery; memory and CPU are the sum of the four nodes. See the Test Environment page for the single-node method and the client.

The four throughput scenarios use MQTT clean sessions (clean_session=true). They measure routing and delivery throughput, and a clean session leaves no state to carry when a node goes down, so those runs say nothing about session migration. That behaviour is measured separately, below.

Throughput is only half of what a cluster has to do. What happens when a node dies under load — whether the survivors keep serving and the victim comes back — is on the Cluster Survivability page.

500K Connections

500,000 clients connecting at 5,000/s, spread evenly over the four nodes: 125,000 each.

Client: AWS c5n.4xlarge, 29 bound source addresses, xmq_scn 1.0.0, no CPU affinity

Broker ▲VersionMessagesAchieved rateAvg latencyCPUAvg RAMMedian RAMMax RAM
EMQX5.8.9500,0004,545/s(below target)1.7ms405% mean / 568% peak (cluster total)6.51 Gb6.77 Gb9.87 Gb (cluster total)
XMQ1.0.0500,0004,545/s(below target)544us104% mean / 130% peak (cluster total)1003 Mb977 Mb1.65 Gb (cluster total)
EMQX
XMQ

Delivery latency over the test

Average latency per interval, by brokerLatency0us352us704us1.1ms1.4ms1.8ms0s20s40s60s80s100selapsed

Broker memory (RSS) over the test

Broker memory (RSS) per intervalMemory (RSS)0 Mb2.50 Gb5 Gb7.50 Gb10 Gb0s20s40s60s80s100selapsed

Latency uses a logarithmic axis: the brokers differ by several orders of magnitude in this scenario, and a linear axis would flatten the faster ones onto the baseline. Hover the chart for per-interval values.

Fan-In 50K

50,000 publishers send one QoS 1 message each second to a cluster-wide shared subscription with 500 subscribers.

Client: AWS c5n.4xlarge, 29 bound source addresses, xmq_scn 1.0.0, no CPU affinity

Broker ▲VersionMessagesAchieved rateAvg latencyCPUAvg RAMMedian RAMMax RAM
EMQX5.8.929,991,99349,986/s1.5ms1835% mean / 2140% peak (cluster total)12.4 Gb13.5 Gb14.23 Gb (cluster total)
XMQ1.0.029,995,08249,991/s465us646% mean / 745% peak (cluster total)413 Mb431 Mb440 Mb (cluster total)
EMQX
XMQ

Delivery latency over the test

Average latency per interval, by brokerLatency0us779us1.6ms2.3ms3.1ms3.9ms0s120s240s360s480selapsed

Broker memory (RSS) over the test

Broker memory (RSS) per intervalMemory (RSS)100 Mb1 Gb10 Gb100 Gb0s120s240s360s480selapsed

Latency uses a logarithmic axis: the brokers differ by several orders of magnitude in this scenario, and a linear axis would flatten the faster ones onto the baseline. Hover the chart for per-interval values.

Fan-Out 250K

Five publishers send 250 messages/second in total to 1,000 subscribers: 250,000 deliveries/second.

Client: AWS c5n.4xlarge, 29 bound source addresses, xmq_scn 1.0.0, no CPU affinity

Broker ▲VersionMessagesAchieved rateAvg latencyCPUAvg RAMMedian RAMMax RAM
EMQX5.8.974,997,654249,992/s1.7ms1198% mean / 1420% peak (cluster total)1.73 Gb1.73 Gb1.73 Gb (cluster total)
XMQ1.0.074,998,001249,993/s1.2ms447% mean / 526% peak (cluster total)210 Mb210 Mb210 Mb (cluster total)
EMQX
XMQ

Delivery latency over the test

Average latency per interval, by brokerLatency0us334us668us1.0ms1.3ms1.7ms0s60s120s180s240selapsed

Broker memory (RSS) over the test

Broker memory (RSS) per intervalMemory (RSS)0 Mb512 Mb1 Gb1.50 Gb2 Gb0s60s120s180s240selapsed

Latency uses a logarithmic axis: the brokers differ by several orders of magnitude in this scenario, and a linear axis would flatten the faster ones onto the baseline. Hover the chart for per-interval values.

Point-to-Point 50K

50,000 publishers and 50,000 subscribers, one topic each, 2 messages/second per publisher: 100,000 messages/second.

Client: AWS c5n.4xlarge, 29 bound source addresses, xmq_scn 1.0.0, no CPU affinity

Broker ▲VersionMessagesAchieved rateAvg latencyCPUAvg RAMMedian RAMMax RAM
EMQX5.8.959,904,18799,840/s944us2257% mean / 2718% peak (cluster total)7.51 Gb8.17 Gb8.34 Gb (cluster total)
XMQ1.0.059,938,67999,897/s497us1283% mean / 1545% peak (cluster total)825 Mb892 Mb900 Mb (cluster total)
EMQX
XMQ

Delivery latency over the test

Average latency per interval, by brokerLatency0us275us549us824us1.1ms1.4ms0s120s240s360s480selapsed

Broker memory (RSS) over the test

Broker memory (RSS) per intervalMemory (RSS)0 Mb2.50 Gb5 Gb7.50 Gb10 Gb0s120s240s360s480selapsed

Latency uses a logarithmic axis: the brokers differ by several orders of magnitude in this scenario, and a linear axis would flatten the faster ones onto the baseline. Hover the chart for per-interval values.

Stateful session migration

This check is separate from the four throughput scenarios and does not use a clean session. A subscriber with clean_session=false and a fixed client id connected to node-c. That node was then killed with kill -9, as in the survivability test: no handover and no lease released. While it was down, 10,000 unique QoS 1 messages were published at 100/s from node-a. The same client id then reconnected to node-d.

MeasuredResult
Messages published while the owner node was down10,000
Delivered after reconnecting to another node10,000
Missing0
Duplicates0
Also received by a live clean subscriber on node-b10,000
ResultPASS

Every message published during the outage reached the persistent session on the node it reconnected to, with no loss and no duplicates. The live subscriber shows that publishing itself continued while the owner was gone. QoS 1 is at-least-once, so a retransmission would have been allowed; none occurred.

Connection-rate headroom (XMQ only)

The 500K scenario above paces 5,000 connections per second, because that is a pace both products sustain and a comparison has to be like for like. This is the other question: how fast can XMQ accept connections before something in the stand gives. The same 500,000 clients, at four rising paces, each on a freshly restarted cluster.

Pace asked forSustainedMedian connectResult
10,000/s10,000/s617 µsheld
20,000/s19,540/s527.9 msat the knee
40,000/s21,552/s6366.5 mssaturated
80,000/s21,436/s9339.0 mssaturated

XMQ holds 10,000/s with sub-millisecond connect latency. At 20,000/s the clients still all connect, but half a second of queue has built up behind them, and above that the stand settles near 21,000/s however fast the scenario asks. Neither the brokers nor the client are the constraint: each broker stays under one core of its eight and the client's 16 vCPUs sit mostly idle. The connection path is: it makes about seven round trips to the shared Redis per connection — 3.5 million commands for 500,000 clients, measured with INFO commandstats — and a round trip to that Redis is about 0.1 ms on this stand. With four listener threads per node, that is roughly 4,500 connections/s per node, or about 18,000 for the cluster, which is where the ladder lands. Redis itself stays near half a core: the cost is the number of round trips, not its CPU. Taking session registration off the connect path, or batching it, is what would raise this ceiling.