Webhook and Data Integration
Webhook and data integration
The webhook extension posts what the broker sees to one HTTP(S) endpoint. It has two inputs - the MQTT messages the broker accepts, and the broker's own log records - and either or both can be sent. Every request carries an HMAC-SHA256 signature when a secret is configured.
It is an extension, not a core feature: a broker with no webhook loaded pays nothing for it. It is built for Linux, FreeBSD and Windows: the socket layer uses Winsock on Windows and the platform's POSIX sockets elsewhere, and TLS is OpenSSL on all of them. On Windows the installed fragment nameslib\xmq-extensions\xmq_webhook.dll; on the other platforms the library is libxmq_webhook.so.
One extension is one endpoint. To send the same stream to several receivers, put a small fan-out service in front of this one; it is a better place for per-receiver filters and credentials than a JSON document typed into a settings field.
Enabling it
The shipped fragment xmq_extensions.d/50-webhook.conf is disabled and has an empty URL, so nothing is posted until both are set. Either edit that file, or switch the extension on in the interface - which writesxmq_extensions.d/90-webhook.conf - and fill in the settings there. Loading, ordering and the Required flag are described on theExtensions page.
A URL that is missing or malformed is refused and logged when the extension starts; it does not start in silence and lose records. The same holds for a plainhttp:// URL while allow_http is false.
What is sent
sources chooses the input. Send messages posts the messages the broker accepted, Send log posts the broker's log records, and Send log and messages posts both, told apart by the source field in the body. There is no "neither": to send nothing, disable the extension.
| Setting | Narrows |
|---|---|
topic_filter | Messages. A list of MQTT filters; a message matching any item is sent.+ and # must each be a whole level, and# the last one. |
max_payload | Messages. Larger bodies are dropped and counted, not truncated. |
log_min_level | Log records. debug, info, warning orerror, the lowest level that is sent. |
log_subjects | Log records. A comma-separated list of the broker's log subjects, for example publish,connect,server_connections; empty means all of them. |
The log input is what makes an unreachable-account or refused-publication incident visible: refusals from the account database, the permission evaluator and the extension host arrive even when no client can connect, because they are records of why a message does not exist.
What the endpoint receives
A POST with Content-Type: application/json andUser-Agent: xmq-webhook/1.2. With no template set, a message arrives as:
{"topic":"sensors/water/1","message":"aGVsbG8=","level":"message"}The payload is base64 because MQTT bodies are binary and JSON strings are not, and a message has no severity of its own, so level is the literalmessage - which is also what tells the two inputs apart at the receiver. A log record arrives in the same three fields: topic empty,message the log text, and level its severity.
A 2xx answer means the record was delivered. The body is not interpreted otherwise. The connection is kept open between records and re-established if the endpoint closes it while idle.
Body templates
Set template when the receiver wants its own envelope. Names written as{{name}} are replaced, and the value is JSON-escaped, so a quote or a newline in a topic cannot break the body:
"template": "{\"text\":\"[{{level}}] {{subject}}: {{message}}\",\"topic\":\"{{topic}}\"}"The names available for a message are id, source,timestamp, topic, qos,retain, client_id, username,payload_base64, message and level. A log record has id, source, timestamp,level, subject and message, withtopic empty. For a message, message andpayload_base64 are the same base64 text.
A name the template does not know is left exactly as written rather than replaced with nothing, so a typo fails at the receiver instead of silently emptying a field. One template serves both inputs, which is why the default one names only fields that exist in both.
Proving the sender
With secret set, every request carriesX-XMQ-Signature: sha256=<hex>, the HMAC-SHA256 of the exact request body keyed with that secret. A receiver should compute the same digest over the bytes it received and compare in constant time; it should reject a request with no signature when one is expected.
secret is a secret setting: the interface masks it and the broker keeps it out of the log. Empty means unsigned requests, which is acceptable only on a link that is already trusted.
headers adds request headers of your own, one Name: valueitem per entry - an authorization token for the receiver, for example. A header that is malformed is refused when the settings are applied.
Configuration
These are the settings of the extension. Each is checked on theExtensions page and again by the broker; the format column is what the value must match as a whole.
| Setting | Type | Default | Meaning |
|---|---|---|---|
url | string | empty | Where the bodies are posted, http:// orhttps://. Empty means the extension has nothing to do. |
sources | choice | Send messages | Send log, Send messages, or Send log and messages. |
template | string | the three-field JSON body | Body to post, with {{...}} names substituted and JSON-escaped. |
headers | list of strings | empty | Extra request headers, one Name: value per item. |
secret | secret | empty | HMAC-SHA256 key for X-XMQ-Signature. Masked in the interface and never logged. |
topic_filter | list of strings | # | Which messages are sent; one filter per item. The interface shows one field per filter, with Add and Remove beside the list. |
max_payload | integer | 1048576 | Largest message body to send, in bytes; larger ones are dropped and counted. 0 means no limit. |
timeout_ms | integer | 5000 | Connect and read timeout of one request, 1 ms to 999999 ms. |
retries | integer | 3 | Attempts after the first one, 0 to 100. |
concurrency | integer | 4 | Requests in flight to the endpoint, 1 to 64. Each is a worker thread waiting for one synchronous reply. |
allow_http | boolean | false | Permit a plain http:// URL. Meant for a receiver on the same machine and for tests, not for a production endpoint. |
log_min_level | choice | error | Lowest log level to send: debug, info, warning or error. |
log_subjects | string | empty | Comma-separated log subjects; empty means all. |
_comment keys in the shipped fragment are not settings and are not edited on the page; name and library identify the extension itself and live in the configuration file.
Delivery, and what is lost
The publish path never waits for the endpoint. An accepted message copies its body into the extension's queue and the broker continues; a slow or unreachable receiver costs deliveries, not broker throughput.
| Answer or failure | What happens |
|---|---|
| HTTP 2xx | Delivered. Counted as sent. |
| HTTP 5xx, 429, timeout, connection failure | Retried, up to retries times, with a backoff that starts at 100 ms and doubles to at most 2 s. |
| Any other 4xx | Refused without a retry: the endpoint would say no again. |
| The queue is full | The newest record is dropped and the drop counter rises. Delivery is at-most-once and a restart loses what has not been sent. |
The extension holds 1024 bodies in its queue, and concurrency workers send them. A worker waits for one synchronous reply, so an endpoint 10 ms away carries about 100 × concurrency bodies a second at best; raiseconcurrency before raising timeouts.
What it is not, and what it costs
- One endpoint, no transformations. The body arrives as it was published, or in the fixed shape the template gives it. There is no rule language, no JSONPath and no per-record routing. Kafka, database and object-store connectors are further extensions on the same message tap.
- No replay and no durable queue. The queues are in memory, bounded by count rather than bytes. With the default
max_payloadof 1 MiB and a stalled endpoint, the queued bodies can be a gigabyte of memory: setmax_payloadto the largest message the deployment really sends. - Base64 adds a third to the body. A 1 MiB payload travels as roughly 1.4 MB of JSON.
- HTTPS is verified. Certificates are checked against the system store, so a private CA has to be installed where the broker can see it.
- A URL set by an administrator can reach internal services. The payload is the message body, so do not point this at an endpoint that logs bodies which carry credentials.
The counters for each input are shown on the Extensions page as messages and log records delivered and dropped. A non-zero drop count means the endpoint or the network is slower than the traffic that matches the filters.
