XMQ MQTT Server – Documentation

Supported operating systems

XMQ is built and packaged for the systems below. Each is a 64-bit x86 build, packaged in the format native to that system.

Other Linux distributions of comparable vintage generally work when built from source, but only the versions listed here are tested and packaged.

Download and install binaries

The only official download location is here. Select the latest SPTK version and your operating system, and download xmq_server package.

Packages are named XMQ-server and install under /usr/localon Linux and FreeBSD. The server, the utilities, the configuration interface, and the load-test scenarios all come from that one package.

The dependencies for Linux include brotli and pcre2 packages that are included in all major Linux distributions. On FreeBSD pkg resolves them from the official repository. The dependencies for Windows are installed with XMQ.

The package file name carries the version and the architecture, so the commands below use a wildcard. Run them in the directory the package was downloaded to.

Debian and Ubuntu

sudo apt install ./xmq-server_*.deb

Fedora and Oracle Linux

sudo dnf install ./XMQ-server-*.rpm

FreeBSD

sudo pkg install ./XMQ-server-*.pkg

Installing through the package manager rather than dpkg -i orrpm -i lets it resolve the shared libraries the package depends on.

What gets installed

On Windows the equivalents live under C:\ProgramData\xmq, with the program files in C:\Program Files\xmq.

Running the server

The package installs a systemd unit, so on Linux the server is managed the usual way:

sudo systemctl enable --now xmq_server
sudo systemctl status xmq_server
sudo journalctl -u xmq_server -f

On FreeBSD the package installs an rc.d script instead:

sudo sysrc xmq_server_enable=YES
sudo service xmq_server start
sudo service xmq_server status

On Windows, xmq_server --install-service registers it as a Windows service, and --uninstall-service removes it.

Useful options when running it by hand:

Setting the administrator password without the interface

A freshly installed server has an administrator with no password, and until one is set the configuration interface answers on 127.0.0.1 only. On a machine with a console that is exactly right. Where there is no console it is a problem, so the password can also be set from the command line:

sudo -u xmq xmq_server --set-password admin

It asks for the password without showing it, and from a pipe reads standard input instead, so the password never appears in the process list or the shell history. On a headless server the alternative is to forward the port —ssh -L 18883:127.0.0.1:18883 the-server — and use the interface as usual. In a Docker container, run it with docker exec -it <container> xmq_server --set-password admin, then restart the container; keep /etc/xmq on a volume and it stays set.

Once the server runs, the configuration interface is reachable on port 18883 by default. On a server that has just been installed it answers on that machine only, at https://127.0.0.1:18883, and the admin account it created has no password: sign in as admin leaving the password empty. The Initial Setup page opens straight away, and the password set there is what opens the interface to the rest of the network.

Environment tuning

A default Linux install is tuned for a few thousand sockets, not a few hundred thousand. Below roughly 10,000 concurrent connections the stock settings are fine and this section can be skipped. Above that, the limits below are the ones that bite, and each of them has been hit in real test runs.

Ready-made files are installed with the package under/usr/local/share/xmq/setup, along with a README describing the symptom each one addresses:

cd /usr/local/share/xmq/setup
sudo install -m 0644 sysctl.d/mqtt.conf       /etc/sysctl.d/mqtt.conf
sudo install -m 0644 sysctl.d/port_range.conf /etc/sysctl.d/port_range.conf
sudo install -m 0644 limits.d/mqtt.conf       /etc/security/limits.d/mqtt.conf
sudo modprobe nf_conntrack
sudo sysctl --system

Then log out and back in, so the nofile limit applies to your shell.

The three that catch people out

• One descriptor per connection, plus a handful. The server idles at 17 descriptors, so a limit of exactly 1,000,000 fails at 999,983 connections. The shipped files set 2,000,000, which leaves room.
• Services do not read limits.d. That file applies to login sessions only. A systemd service gets DefaultLimitNOFILE instead, commonly 524288. The XMQ unit sets LimitNOFILE explicitly for this reason; any unit you write yourself must do the same.
• A ulimit -n in your shell profile wins. If~/.profile or ~/.bashrc lowers it, an unprivileged process cannot raise it again, and the limit files appear to have no effect.

Both ends need this. Every limit here is per host, and tuning only the server while leaving the client machine at defaults is the most common mistake. Thepreflight.sh script in /usr/local/share/xmq checks a host and exits non-zero if a required setting is missing:

./preflight.sh                          # check this host as a client
./preflight.sh -H broker -r server -p 1883   # check a remote server host

Receive packet steering on a single-queue network card

A network card with one receive queue delivers every incoming packet to one CPU, and the kernel runs the whole IP and TCP receive path there. Under heavy load that CPU reaches 100% while the others idle, the card starts dropping frames, and clients retransmit into backoff until the server closes their sessions for a keep-alive timeout - although every one of them was sending. On a test host with such a card, a 100,000 messages a second run lost 255,891 frames a second this way and never finished. Linux receive packet steering (RPS) spreads the work over other CPUs, and with it the losses fell to zero and the run completed.

XMQ checks this at start-up. When a network interface its listeners use has a single receive queue, no steering, and the machine has more than two CPUs, the log says so and links here. Enable RPS when all three of these hold:

• The card has one receive queue. ls /sys/class/net/eth0/queues/lists only rx-0. Cards with several queues already spread their interrupts across CPUs and do not need RPS.
• One CPU is pinned under load. While clients are connected and busy,mpstat -P ALL 1 shows %soft near 100 on one CPU. Checking after the load has stopped shows nothing.
• The card is dropping frames. The drop counters inethtool -S eth0 - rx_no_buffer_count,rx_missed_errors or rx_fifo_errors, depending on the driver - grow while the load runs.

With only the first condition, leave it off: moving packets between CPUs costs a little at low rates. To enable it, write a mask of the CPUs to use, and size the flow tables so that each packet goes to the CPU where its socket is read:

IF=eth0
echo 70    | sudo tee /sys/class/net/$IF/queues/rx-0/rps_cpus      # hex mask: CPUs 4, 5 and 6
echo 32768 | sudo tee /proc/sys/net/core/rps_sock_flow_entries
echo 32768 | sudo tee /sys/class/net/$IF/queues/rx-0/rps_flow_cnt

Choose CPUs that are idle under load, not the one that receives the card's interrupt (grep eth0 /proc/interrupts shows which), and on the card's NUMA node. The settings do not survive a reboot. Keep them with a sysctl file and a udev rule:

# /etc/sysctl.d/60-rfs.conf
net.core.rps_sock_flow_entries = 32768

# /etc/udev/rules.d/60-rps.rules
ACTION=="add", SUBSYSTEM=="net", KERNEL=="eth0", ATTR{queues/rx-0/rps_cpus}="70", ATTR{queues/rx-0/rps_flow_cnt}="32768"

Or with a systemd unit, which needs no interface name: it spreads every receive queue of every wired card over all CPUs at boot. It suits a small machine with one card - a cluster node on such a host answered in 3.2 ms without it and in 0.42 ms with it, at 50,000 messages a second.

# /etc/systemd/system/rps.service
[Unit]
Description=Spread received packets over all CPUs (RPS) on NICs with a single receive queue
After=network-pre.target
Before=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'mask=$(printf "%%x" $(( (1 << $(nproc)) - 1 ))); for q in /sys/class/net/e*/queues/rx-*/rps_cpus; do echo $mask > $q; done'
ExecStop=/bin/sh -c 'for q in /sys/class/net/e*/queues/rx-*/rps_cpus; do echo 0 > $q; done'

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now rps.service
cat /sys/class/net/*/queues/rx-0/rps_cpus      # every CPU's bit set
sudo systemctl disable --now rps.service       # back to one CPU

Other systems have their own equivalents, which XMQ does not check. On FreeBSD it is netisr: net.isr.maxthreads and net.isr.bindthreads in/boot/loader.conf, with net.isr.dispatch=deferred. On Windows it is receive side scaling, set with Set-NetAdapterRss, and available only when the card's driver supports several queues.