XMQ MQTT Server – Setting up a Cluster

Setting up a Cluster

Available since XMQ 1.0.0.

A cluster is a set of XMQ nodes that share one storage (Redis) and one accounts database (PostgreSQL or MySQL), so that a client connecting to any node sees the same sessions, the same accounts and the same retained messages. A node joins a cluster by invitation: an administrator of the running cluster declares the new node by its name and TLS address, invites it, and the invited node reviews and accepts - the node then restarts into the cluster.

The steps below set up two nodes, node-a and node-b, on two hosts. Everything is done from the configuration interface: a cluster is declared by an administrator there, not by editing a file on each node.

Prerequisites

• XMQ 1.0.0 or newer, installed on both nodes - seeDownload and install binaries.
• PostgreSQL (or MySQL) reachable from both nodes - the shared configuration and the shared accounts. The table xmq_node holds what an administrator edits: a row is a node, a column is a setting. A node that cannot reach the database does not start.
• Redis reachable from both nodes - membership, leases, sessions and retained messages. Any database number works; the cluster keeps its keys under cluster: and its sessions undersession_<client_id>.
• Nothing else. The TLS certificates the nodes link to one another with are created by the nodes themselves, one per node, during the initial setup - there is no certificate to obtain or copy before starting.

1. Install and set up node A

Install the package on the first host, start the server, and open the configuration interface. A server that has never been set up answers on the loopback address only, and its admin account has no password yet: sign in with the password field left empty, and the Initial Setup page opens.

# On node-a, after installing the package:
sudo systemctl enable --now xmq_server

# The interface is on the loopback address until a password is set. From the
# host itself, or through a tunnel:  ssh -L 18883:127.0.0.1:18883 node-a
#   -> https://127.0.0.1:18883

On the Initial Setup page fill in: the administrator password, the node name (node-a), the node host - the address the other nodes reach this one at, be it a DNS name or an address - the MQTT, MQTT+SSL and interface ports, and, if Redis is already running, its address. Leaving the Redis address empty is fine: it can be given later, on the Dashboard.

This page is also where the node's certificate is created: a self-signed certificate of the node's own, made out to the node host entered above, is written to /etc/xmq/certs/node.crt together with its key, and both the configuration interface and the MQTT+SSL listener are pointed at that pair. One certificate per node, never copied between them - it is the node's identity on the cluster's links.

Verify: the interface now answers on the node host, with the new certificate:

openssl s_client -connect node-a:18883 </dev/null 2>/dev/null     | openssl x509 -noout -subject -dates
# subject= ... CN=node-a   (the node host given on the setup page)

2. Install and set up node B

The same on the second host: install, start, open the interface through the loopback or a tunnel, and fill the Initial Setup page with node B's own name (node-b), node host, and ports. Its certificate is created the same way, made out to its host.

Verify: the same command against node B returns a certificate namingnode-b. The two nodes now have two different certificates - check with openssl x509 -noout -fingerprint -in /etc/xmq/certs/node.crt on each, and see that the two differ.

3. Give every node the same cluster password

Each installation creates its accounts with an internal clusteraccount whose password was generated at random. It is the account one node signs in to another with on every cluster link and on the invitation, and it has to be the same on every node of the cluster: set it to one value of your own choosing.

On each node open the Users page, select the cluster account, and set its password (the dialog has a Set Password action). Use the same password on both nodes.

Verify: the Users list shows the cluster account as enabled on both nodes. The password itself is never shown back by the interface - it is stored as a verifier.

4. Trust each node's certificate on the other

A cluster link is TLS with peer verification, and so is the invitation: node A verifies node B's certificate, and node B verifies node A's. The nodes do not trust each other until they are told to - by the certificate itself, which is its own authority.

On node A open SSL Keys, and under Trusted Nodes add node B: give it the name node-b and paste the content of node B's certificate (/etc/xmq/certs/node.crt). Then do the same on node B with node A's certificate.

# On node-b, to get the text to paste into node-a's interface:
cat /etc/xmq/certs/node.crt

Verify: each node's SSL Keys page lists the other under Trusted Nodes, and the certificate is on disk:

ls /etc/xmq/certs/peers/
# node-b.crt   (on node-a)   /   node-a.crt   (on node-b)

5. Create the shared database

On the PostgreSQL host, create a role and a database for the cluster. Both nodes will be pointed at it: node A by the interface, node B by the invitation - so the address is entered once.

sudo -u postgres psql -c "CREATE ROLE xmq LOGIN PASSWORD 'yourpassword'"
sudo -u postgres createdb -O xmq xmq_cluster

# Verify:
psql "postgresql://xmq:yourpassword@db-host/xmq_cluster" -c '\dt'
# (no tables yet - the node creates them on its first start)

6. Make node A the first member of a cluster

On node A open the Dashboard and switch the mode from Standalone toCluster. The dialog asks for the two settings every node of the cluster shares: the storage, as a Redis URI such asredis://redis-host:6379/5, and the accounts database, as the address created in step 5 (postgresql://xmq:yourpassword@db-host/xmq_cluster).

Both are written into the node's own configuration - the bootstrap file keeps the SQL URI and the interface port, and everything else a cluster shares comes from the database - and the node restarts. On that start it moves its accounts into the database, creates its row there, and takes its place in the cluster as its first member and its coordinator. The storage and the database are checked before anything is written: an address that does not answer, or a storage that already belongs to another cluster, is refused with the reason.

Verify: the Cluster Configuration page now shows the node's own row, state Clustered, and names node-a as the coordinator. And in the database:

psql "postgresql://xmq:yourpassword@db-host/xmq_cluster"     -c 'select name, state, cluster_enabled from xmq_node'
# node-a | Standalone | t
# (Clustered is computed from the storage, not stored - the row keeps what is written)

7. Declare and invite node B

On node A's Cluster Configuration page press Add and declare node B: the name node-b and the TLS address other nodes reach it at -node-b:8883, the MQTT+SSL port of node B. Nothing is contacted yet; the row appears as Not Clustered.

Select the row and press Invite. The dialog asks for the password of thecluster account - the one set in step 3. Node A then connects to node B over TLS as that account (which is where the trusted certificate from step 4 is needed), asks it who it is, and publishes the invitation.

Verify: node B's own Cluster Configuration page now shows the invitation: who invited it, and the storage and accounts database the cluster works against. On node A the row of node B says Invited.

8. Accept the invitation on node B

On node B press Review and Accept and read the confirmation: it names everything the accept changes - persistence switched to the cluster's storage, the accounts moved to the cluster's database, the cluster enabled, the node restarting. Local sessions, queued messages and local accounts are not carried over.

Accepting writes the new configuration and ends the process with the restart code; the package's service unit is Restart=always, so systemd brings the node back on the new configuration by itself. On that start node B joins the shared database and the shared storage and takes its place in the cluster.

Verify: on node A's Cluster Configuration page both rows now sayClustered, and both are alive:

# On the Redis host, if you want to see the membership itself:
redis-cli -n 5 zrange cluster:members 0 -1
# (two GUIDs - the two nodes)

9. Check that the cluster carries traffic

Publish a retained message through one node and read it through the other: a retained message is synchronised across the cluster, so a subscriber on node B receives what was published on node A.

# Through node A:
xmq_pub -h node-a -p 1883 -u admin --password admin -t cluster/test     -m "hello from node-a" -q 1 -r

# Through node B:
xmq_sub -h node-b -p 1883 -u admin --password admin -t cluster/test -C 1 -W 5

If the message arrives, the cluster is whole. A client that then connects to either node for an ordinary session is served by that node, and the other nodes forward to it whatever its subscribers are subscribed to.