MQTT Performance Tests: Test Environment

Test Environment

All published XMQ benchmark results are produced on AWS EC2, using a dedicated server instance and a separate client instance. Load generators are never run on the broker host: a client competing for the same cores inflates latency and makes the result impossible to attribute. Every figure quoted in the test pages was measured with the client on its own instance.

AWS configuration

• Single availability zone. Both instances run in us-east-1a. Cross-AZ traffic would add roughly a millisecond of round trip on its own, which is several times the latency being measured.
• Cluster placement group. Both instances are members of the xmq-load-test cluster placement group, packing them onto the same low-latency network segment for consistent, high bandwidth between them.
• Processor. Intel Xeon Platinum 8124M (Skylake-SP) at 3.0 GHz base.
• Multiple source addresses. The client binds connections across 30 secondary private IP addresses. A single address exhausts its ~64K ephemeral ports well before the connection counts these tests use, and port allocation slows sharply long before that limit is reached.
• Operating system. Ubuntu 26.04 LTS, kernel 7.0.0-aws.

Broker tuning

Every broker is tuned before measurement, not run at its packaged defaults. A benchmark that leaves any broker misconfigured proves nothing.

• EMQX is configured following the official EMQX performance tuning guide, and the OS settings below are taken from it. Its Erlang VM settings were left at the shipped values after testing confirmed they are the right choice at these message rates: raising the scheduler busy-wait threshold (+sbwt) improves latency below roughly 20K messages per second but costs throughput above it, and reducing the scheduler count to match physical cores was substantially worse.
• Mosquitto runs with set_tcp_nodelay true, unlimited max_inflight_messages and max_queued_messages, persistence disabled, and a raised file descriptor limit. Mosquitto's defaults for the two queue limits throttle delivery at high publisher counts, and its default leaves Nagle's algorithm enabled.
• FlashMQ runs with tcp_nodelay true, thread_count 16 (one worker per vCPU, which is also what its auto-detection chooses on this host), and log_level warning with subscription and publish logging off — at the lower levels it logs every connection and every publish, which at these rates measures the log rather than the broker. Its max_qos_msg_pending_per_client and max_qos_bytes_pending_per_client are raised from their 512-message and 64 kB defaults to 65535 and 256 MB: at the defaults QoS 1 traffic is dropped when a subscriber falls behind, which would be recorded as throughput the broker never delivered. This is the same reasoning behind the unbounded queues given to Mosquitto.

Kernel settings

Applied identically to broker and client. The system tuning is similar to that described in the EMQX performance tuning guide, and the same host configuration is used for every broker tested, so none is advantaged by it.

Transparent huge pages are left at madvise, so they are used only where explicitly requested rather than assembled in the background, and swap is disabled. Both avoid latency spikes that would otherwise appear as unexplained outliers.

Process limits

The fs.nr_open and fs.file-maxsettings above are only the system-wide ceiling. The per-process limit is separate, and it is the one a run actually hits — a connection costs one descriptor, so the limit has to exceed the connection count with room for listeners, the event reactor and any database sockets. Both hosts set:

Two details are worth stating because both produce failures that look like broker faults rather than host misconfiguration. A per-user ulimit -n in a shell profile sets the hard limit as well as the soft one, so a value below the figures above silently caps every subsequent run and cannot be raised again by an unprivileged process. And a limit equal to the connection count is not enough: the broker's own listener and reactor descriptors have to fit too, so a run targeting exactly the limit stops a few connections short and reports them as stalled handshakes.

One further setting applies only where connection tracking is active — a host running Docker or libvirt, which load nf_conntrack whether or not anything is being filtered. Each connection then consumes a tracking entry against net.netfilter.nf_conntrack_max, whose RAM-derived default is around 262K. Beyond it the kernel drops packets silently, which presents as connect timeouts and unreachable-server errors from the client while the broker itself sits idle. Where this applies, the MQTT ports are exempted from tracking outright rather than the maximum simply raised, which also keeps the per-packet tracking lookup out of the path being measured.

Load generator and scenarios

Every figure on these pages is produced by xmq_scn, the scenario runner shipped as part of the XMQ server installation — the same binary that is on any machine where XMQ is installed, not a private harness. It drives every broker identically: nothing in it is XMQ-specific, it speaks plain MQTT 3.1.1/5.0, and the broker under test is chosen with nothing more than a host and port. See the MQTT Test Suite page for a full description of xmq_scn — its scenario format, command-line options and output.

A test is a JSON scenario file: a type (Point-To-Point, Fan-In, Fan-Out or Connections), publisher and subscriber counts, topic count, QoS, payload size, publish or connection rate, and duration. Because the broker is only a host and port, the identical file runs against each broker in turn.

The supplied scenarios deliberately reproduce the Basic and Enterprise sets from EMQX's published broker benchmark, so results here can be read against theirs rather than only against each other. The file names encode the parameters:

Additional intermediate scenarios (for example point-to-point at 10K, 20K and 35K pairs, and connection tests at 100K and 500K) fill in the gap between the two sets, where a broker's behaviour often changes.

Around xmq_scn sit three shell scripts, so a run is reproducible rather than a remembered command line. run_load_test.sh is the entry point: it confirms the broker is reachable, optionally applies the client-side kernel tuning above, brings up the secondary IP addresses, then invokes xmq_scn with the scenario and any overrides. make_ip_addresses.shassigns the secondary addresses, and a per-environment init_environment_*.sh supplies the broker host, port and client subnet through environment variables — so the scenario files themselves stay free of any site-specific addressing and are used unmodified everywhere.

For the fuller statement of the conditions these scenarios come from, see EMQX's open MQTT benchmark and the Open MQTT Benchmark Suite results the scenario definitions are taken from — the file names above follow that repository's own naming, so each scenario here has a direct counterpart there. Both describe a single c5.4xlarge (16 cores, 32 GB, Ubuntu 22.04) driven by XMeter, comparing EMQX 4.4.16 and 5.0.21, Mosquitto 2.0.15 and NanoMQ 0.17.0.

Those results are from 2023 and should be read as a definition of the scenarios, not as a current baseline. Every broker version in them is by now several releases old, which is why the brokers compared on these pages are re-run here at current versions rather than quoted from that publication.

Their published latencies could not be reproduced here. For the 50K point-to-point scenario the suite reports 1.68 ms average for EMQX 4.4.16; the same scenario, re-run for these pages, puts EMQX at 53 ms — roughly thirty times higher.

The hardware is not the explanation. Their c5.4xlarge and the c5n.4xlarge used here are both 16 vCPU Xeon Platinum 8124M (Skylake-SP); the n variant differs only in carrying more memory and 25 Gbps of network against 10, so the runs here had, if anything, the more capable machine. What does differ is the broker build — their EMQX 4.4.16 and 5.0.21 against a later 5.x release, installed from the vendor's own package repository — and the load generator, XMeter against xmq_scn, which also means a different point at which latency is timestamped. Either could account for some of a gap this size; neither is verifiable from the published material, which does not state where its load generator ran.

This is not offered as a correction to their figures, and the scenario definitions remain useful regardless. It is the reason every number on these pages is a measurement taken here, with its conditions stated, rather than a citation of someone else's. Where the two do agree is on Mosquitto: their results likewise record it failing to reach the target rate in this scenario, settling at 37.3K messages/second against the 50K offered.

Two further differences are worth keeping in mind when reading the two side by side: the results here use a separate instance for the load generator, and they are reported as per-interval averages across the run rather than as one figure per test.

Method

• Identical client and scenario for every broker. Only the broker changes between runs. Where a broker's own published figures differ from those measured here, this is the difference most likely to explain it: published benchmarks frequently omit how load was generated.
• Results reported as interval averages across the whole run, not as a single figure, so that latency drifting upward over time is visible rather than averaged away. Run duration is stated on each test page, since it varies by test.
• Brokers restarted between runs so that session state from a previous run cannot carry into the next.