On the MQTT 5.0 Message Expiry Interval

There's a neat little feature in the MQTT 5.0 spec called Message Expiry (or Message Expiry Interval).

It has been explained many times on the internet already. It allows publishers to add an expiry interval to any message they send. Sometimes, we have seen a misconception about the feature though. And the misconception is that Message Expiry was missing from the MQTT 3.1.1 protocol, and now that it's there in 5.0 it will supersede and replace retained messages and other persistent session features. Well, not that fast.

You'll most probably know Message Expiration (or TTL, time-to-live) from traditional enterprise messaging systems. The canonical example is here: Message Expiration Pattern.

The Traditional Message Expiration Pattern vs. MQTT

The application for this pattern is mostly this:

  • First, you want a consumer to process a job (Application Message) within a specific time window.
  • Second, you want to forward the Application Message to a Dead letter Queue in case the consumer couldn't handle it in time. From the Dead letter Queue, your backend system can handle the message and re-trigger the job or any other needed actions.

Now, MQTT has no queue concept, and it certainly has no Dead letter Queue concept! So we have quite a different situation from the beginning. You can't just configure Dead letter Queues (or Dead letter topics). That's also the reason, by the way, that MQTT has no "Queue TTL" in addition to Message TTL, like other systems. (note that it has Session Expiry, which is again, a completely different thing).

MQTT 5.0 explicitly states that "dead lettering" is not part of the spec:

"Where a packet is discarded without being sent, the Server could place the discarded packet on a 'dead letter queue' or perform other diagnostic action. Such actions are outside the scope of this specification."

Does this leave us with a half-baked useless feature?

No, not at all. First, if you need some Dead letter functionality, you can still implement this with a client-side application component (or broker plugin).

And second, there's obviously still applications that can benefit from simple time-based expiry, without Dead letter queues. Imagine you have mobile apps that connect to an MQTT broker after you sent them a push notification. You could send them a time-based coupon offer in an Application Message. If the coupon expires, there's no need for latecomers to get that message at all.

Using Message Expiry will also help you manage broker state more effectively. If you know that messages won't be needed forever, you can tell the broker and allow it to store and manage fewer messages in offline queues.

The Nature of MQTT (SCADA)

Reflecting a little bit deeper on the nature of MQTT, you can see its differences to traditional EAI and protocols like AMQP, even in a little feature like Message Expiry. To understand this, you need to look into where MQTT is coming from, namely supervisory control and data acquisition (SCADA).

In SCADA, the main question is whether a SCADA host knows the state of the field devices it manages. Are the reported values "good", or "stale" (unknown)? What the SCADA host wants is LGVs (last good values). And it needs to be sure about an LGV, even when the device has disconnected multiple times, even when the device hasn't sent a status update for days. The purpose of all the session state persistence is exactly that: ensure the LGV and allow report-by-exception style messaging instead of regularly polling for current field value status. Features like the retained message feature (always retaining the last incoming message) might be enough for cases like that.

On the other hand, there's no point giving an Application Message an expiry interval, when your use case is the exact opposite: avoiding stale values. (or there's at least not much of an advantage compared to merely polling for values).

If this doesn't make too much sense to you, don't worry. My point here is that it's often useful to remember where MQTT is coming from. Sometimes we assume messaging is all the same, and MQTT just somehow the "easiest version" of it.

Let's look at a quick example using the Mosquitto command line tools. Using the Mosquitto command-line tools, you could say:

mosquitto_pub -h localhost -p 1883 -q 1 -t 2020 -m Happy New Year! -V 5 --property publish message-expiry-interval 40

This is only a minimal example, but note how you add the message-expiry-interval. You add it as a --property, stating it's a PUBLISH property. You then add the message-expiry-interval keyword and the actual lifespan of the message in seconds. Note that from the publisher side, this only works when you enforce protocol version 5.0 (using -V 5).

The cool thing is that your consumers can still be using version 3.1.1, and benefit from the Message Expiry feature. VerneMQ completely handles this. A subscriber with a stateful 3.1.1 session, would not receive our Happy New Year message if it wouldn't get online in those 40 seconds:

mosquitto_sub -h localhost -p 1883 -u test -P test -t 2020 -q 1 -i SUBSCRIBER -c

Note that while the MQTT specification is relatively precise, there currently might be slightly different understandings on some minor aspect of message expiry among broker implementors. If a client with a clean session and QoS 0 comes online in that 40-second window, does it get the Application Message or not? There's also some quite esoteric bits and parts in the spec. For instance, you can also set an expiry interval for the Will Message:

"If present, the Four Byte value is the lifetime of the Will Message in seconds and is sent as the Publication Expiry Interval when the Server publishes the Will Message."

Go figure out a use case for that.

We try to be pragmatically implementing those features, but we also listen to users and use cases.

So, let us know how you're going to use the Message Expiry feature!