vmq-diversity - The new Plugin builder toolkit for VerneMQ

You might wonder where we've been the last couple of weeks as we haven't released a blog post since quite a while. Shame on us! Development certainly hasn't stalled and we're progressing quickly. As an example, let's take a closer look at our new plugin called vmq-diversity, which essentially brings scripting support for VerneMQ. We'll also summarize some of the many important improvements we made during the last months. But let's start with the big announcement of vmq-diversity.


vmq-diversity is a new VerneMQ plugin that allows you to implement custom VerneMQ plugins using the Lua scripting language. We run a new publicly available VerneMQ broker under public.vernemq.com with open MQTT (1883) and MQTT over Web-sockets (8000) endpoints. This public instance is for testing purposes only and we'd appreciate if you treat it respectfully. For the remaining of this blog post, we'll look at the auth-handling Lua script we use for the public instance, to give an example of what you can do with embedded Lua.

Start with vmq-diversity right away, and read the plugin development docs.


One of the initial design goals of VerneMQ was and still is that it comes with sane defaults but can be adapted to your needs whenever required. With vmq-diversity we want to go one step further, and open up plugin development to folks not familiar with Erlang. As a simple, yet powerful scripting language, Lua lets you have a go at your own VerneMQ plugins. No excuses anymore! ;)

Let's look at a simple example:

function register_callback(reg)
    if reg.username == "Joe" and reg.password == "secret" then
        return true
    return false

hooks = {
    auth_on_register = register_callback

Of course all other hooks described in our plugin development guide can be implemented in Lua now. But vmq-diversity has even more included.

In order to write powerful Lua scripts, we need a way to interface with the outside world. vmq-diversity comes with several data providers allowing a Lua plugin to talk to:

  • PostgreSQL
  • MySQL
  • MongoDB
  • Redis
  • Erlang Term Storage (ETS)

But it also comes with a simple HTTP, Json, and logging library. There's a lot of stuff for more blog posts in those details. For the moment we recommend to checkout the vmq_diversity/tests folder and our vmq-diversity documentation.

In order to show you a larger example I have to talk a bit about where we've been and what we've been doing during the last months:

Where have we been? Are we still alive?

As mentioned we were quite busy finishing the work on vmq-diversity but we also invested quite some time in finding and fixing bugs. Most of them were reported by you guys, which we heavily appreciate! You're awesome! Several other interesting improvements (e.g. message leak discoveries in our message store, ouch ;)) were made after some larger test runs. We also leveled up our testing gears with a project called vmq_mzbench, which we've released as well and will be covered here in one of our next posts, allowed us to make performance testing fun again. You see we were quite busy, and we think the time was well invested, as the backlog for the version 1.0 release gets smaller every day.

Part of our testing strategy is to offer a public VerneMQ node for small tests. This node can be reached over MQTT (1883) and Web-sockets (8000) under public.vernemq.com. The node is fairly limited and we'd appreciate if you treat it respectfully. Also, please don't trust this node to be up 100%, as we will roll out experimental stuff on it frequently. And yes, we're using vmq-diversity on it which brings me back to the main focus of this blog post.

The public VerneMQ instance

The public instance should mainly give users new to MQTT a chance to play around without having to install a broker. This has been requested multiple times. For the public endpoint we have developed a simple Lua script that covers the following configuration/features:

  • Allow anonymous clients
  • Log every client interaction
  • Don't allow subscriptions on $SYS and #
  • Allow everything for user with superuser privileges
  • Limit the amount of connections per IP address

In order to make this all work we've implemented the following hooks:

Let's start with captain obvious, auth_on_register

function auth_on_register(reg)
    local nr_of_conns = current_conns(reg.addr)
    if not (nr_of_conns < max_conns_per_ip) then
        log.warning("CAN'T REGISTER: ip: " .. reg.addr .. " user: "..reg.username .. " client: ".. reg.client_id .. " due to max_conns_exceeded")
        return false

    local e = {} -- element
    e[reg.addr] = nr_of_conns + 1
    e[reg.client_id] = reg.addr -- index item, supposed to be unique
    kv.insert(tab, e) -- inserts two objects in ets

    local pwd = superusers[reg.username]
    if pwd then
        -- superuser tries to login
        if reg.password == pwd then
            log.info("REGISTER SUPERUSER: ip: " .. reg.addr .. " user: "..reg.username .. " client: ".. reg.client_id)
            return true
            -- a superuser with wrong password is always denied
            return false
    log.info("REGISTER: ip: " .. reg.addr .. " user: "..reg.username .. " client: ".. reg.client_id)
    return true

We're using a kv and a log library as well as a custom function current_conns. The kv library gives us access to a simple key-value database using Erlang ETS, which we use to store the connection tracking counters. The database is initialized like this:

local tab = "client-counters"
kv.ensure_table({name = tab})

The current_cons function implements the access to the connection tracking information.

function current_conns(ip)
    local ret = kv.lookup(tab, ip)
    if #ret > 0 then
        return ret[1]
        return 0

The connection tracking is very simple. It is one key/value table that holds two types of elements:

  • Key = IP Address, Value = Connection Counter
  • Key = MQTT Client Id, Value = IP Address

The superusers table is initialized with username/password pairs of the users we give superuser privileges, similar to this:

superusers = {}
superusers["bob"] = "bobsecret"
superusers["alice"] = "alicesecret"

Once a client disconnects we have to decrement the connection counters. This is done in the cleanup_counters function and called from the on_client_offline and on_client_gone callbacks.

-- clean up counters if required
-- returns the new connection count for
-- this IP address
function cleanup_counters(sub)
    local ip = kv.lookup(tab, sub.client_id)[1]
    local nr_of_conns = current_conns(ip)
    kv.delete(tab, sub.client_id)
    if nr_of_conns == 1 then
        kv.delete(tab, ip)
        return 0
        local new_nr_of_conns = nr_of_conns -1
        local e = {} --element
        e[ip] = new_nr_of_conns
        kv.insert(tab, e)
        return new_nr_of_conns

The last missing piece is the auth_on_subscribe function that is mainly use to distinguish subscriptions made from 'normal' clients from the ones belonging to 'superusers'. We want to do this as we don't like to expose the $SYS topic tree (useful for monitoring purposes) to everyone. We also want to disallow subscribing to every published message via the root wildcard #, even if this would probably be fun to watch...

Other than that, we allow pretty much everything. The callback is implemented like this:

function auth_on_subscribe(sub)
    local logline = "SUBSCRIBE: client: "..sub.client_id
    for k, s in pairs(sub.topics) do
        local topic = s[1]
        local qos = s[2]
        if (string.sub(topic, 1, 4) == "$SYS") or (topic == "#") then
            -- check whether we are superuser
            if not superusers[sub.username] then
                -- we don't allow
                logline = logline .. " topic: " .. topic .. " qos: " .. qos .."\n"
                log.warning("NOT-ALLOWED-" .. logline)
                return false
        logline = logline .. " topic: " .. topic .. " qos: " .. qos .."\n"
    return true

That's it, we've now covered most of the Lua script that is deployed using vmq-diversity on our public.vernemq.com broker instance. I hope you were able to follow us through those few lines of Lua code. And if not, please don't hesitate to get in touch.

Give it a try and head over to the docs.

Some limitations we should mention at this point:

vmq-diversity uses an Erlang based implementation of Lua that can be found here, which provides a subset of the official Lua distribution. While most of the pure Lua modules should work, the ones relying on C libraries e.g. luasockets won't work here. Another limitation is performance, as a recognisable overhead exists when VerneMQ has to call into a Lua plugin.

Thanks for reading!

Your VerneMQ Mission Engineers!