You might wonder where we've been the last couple of weeks as we haven't released a blog post since quite a while. Shame on us! Development certainly hasn't stalled and we're progressing quickly. As an example, let's take a closer look at our new plugin called vmq-diversity, which essentially brings scripting support for VerneMQ. We'll also summarize some of the many important improvements we made during the last months. But let's start with the big announcement of vmq-diversity.
vmq-diversity is a new VerneMQ plugin that allows you to implement custom VerneMQ plugins using the Lua scripting language. We run a new publicly available VerneMQ broker under public.vernemq.com with open MQTT (1883) and MQTT over Web-sockets (8000) endpoints. This public instance is for testing purposes only and we'd appreciate if you treat it respectfully. For the remaining of this blog post, we'll look at the auth-handling Lua script we use for the public instance, to give an example of what you can do with embedded Lua.
Start with vmq-diversity right away, and read the plugin development docs.
One of the initial design goals of VerneMQ was and still is that it comes with sane defaults but can be adapted to your needs whenever required. With vmq-diversity we want to go one step further, and open up plugin development to folks not familiar with Erlang. As a simple, yet powerful scripting language, Lua lets you have a go at your own VerneMQ plugins. No excuses anymore! ;)
Let's look at a simple example:
function register_callback(reg)
if reg.username == "Joe" and reg.password == "secret" then
return true
end
return false
end
hooks = {
auth_on_register = register_callback
}
Of course all other hooks described in our plugin development guide can be implemented in Lua now. But vmq-diversity has even more included.
In order to write powerful Lua scripts, we need a way to interface with the outside world. vmq-diversity comes with several data providers allowing a Lua plugin to talk to:
But it also comes with a simple HTTP, Json, and logging library. There's a lot of stuff for more blog posts in those details. For the moment we recommend to checkout the vmq_diversity/tests folder and our vmq-diversity documentation.
In order to show you a larger example I have to talk a bit about where we've been and what we've been doing during the last months:
As mentioned we were quite busy finishing the work on vmq-diversity but we also invested quite some time in finding and fixing bugs. Most of them were reported by you guys, which we heavily appreciate! You're awesome! Several other interesting improvements (e.g. message leak discoveries in our message store, ouch ;)) were made after some larger test runs. We also leveled up our testing gears with a project called vmq_mzbench, which we've released as well and will be covered here in one of our next posts, allowed us to make performance testing fun again. You see we were quite busy, and we think the time was well invested, as the backlog for the version 1.0 release gets smaller every day.
Part of our testing strategy is to offer a public VerneMQ node for small tests. This node can be reached over MQTT (1883) and Web-sockets (8000) under public.vernemq.com. The node is fairly limited and we'd appreciate if you treat it respectfully. Also, please don't trust this node to be up 100%, as we will roll out experimental stuff on it frequently. And yes, we're using vmq-diversity on it which brings me back to the main focus of this blog post.
The public instance should mainly give users new to MQTT a chance to play around without having to install a broker. This has been requested multiple times. For the public endpoint we have developed a simple Lua script that covers the following configuration/features:
$SYS and # In order to make this all work we've implemented the following hooks:
auth_on_register implements a simple connection tracking mechanism and authenticates superusers auth_on_publish just logs the publish action auth_on_subscribe distinguishes between 'normal' users and superusers for authorizing subscriptions on_client_gone used for connection tracking on_client_offline used for connection tracking Let's start with captain obvious, auth_on_register
function auth_on_register(reg)
local nr_of_conns = current_conns(reg.addr)
if not (nr_of_conns < max_conns_per_ip) then
log.warning("CAN'T REGISTER: ip: " .. reg.addr .. " user: "..reg.username .. " client: ".. reg.client_id .. " due to max_conns_exceeded")
return false
end
local e = {} -- element
e[reg.addr] = nr_of_conns + 1
e[reg.client_id] = reg.addr -- index item, supposed to be unique
kv.insert(tab, e) -- inserts two objects in ets
local pwd = superusers[reg.username]
if pwd then
-- superuser tries to login
if reg.password == pwd then
log.info("REGISTER SUPERUSER: ip: " .. reg.addr .. " user: "..reg.username .. " client: ".. reg.client_id)
return true
else
-- a superuser with wrong password is always denied
return false
end
end
log.info("REGISTER: ip: " .. reg.addr .. " user: "..reg.username .. " client: ".. reg.client_id)
return true
end
We're using a kv and a log library as well as a custom function current_conns. The kv library gives us access to a simple key-value database using Erlang ETS, which we use to store the connection tracking counters. The database is initialized like this:
local tab = "client-counters"
kv.ensure_table({name = tab})
The current_cons function implements the access to the connection tracking information.
function current_conns(ip)
local ret = kv.lookup(tab, ip)
if #ret > 0 then
return ret[1]
else
return 0
end
end
The connection tracking is very simple. It is one key/value table that holds two types of elements:
The superusers table is initialized with username/password pairs of the users we give superuser privileges, similar to this:
superusers = {}
superusers["bob"] = "bobsecret"
superusers["alice"] = "alicesecret"
Once a client disconnects we have to decrement the connection counters. This is done in the cleanup_counters function and called from the on_client_offline and on_client_gone callbacks.
-- clean up counters if required
-- returns the new connection count for
-- this IP address
function cleanup_counters(sub)
local ip = kv.lookup(tab, sub.client_id)[1]
local nr_of_conns = current_conns(ip)
kv.delete(tab, sub.client_id)
if nr_of_conns == 1 then
kv.delete(tab, ip)
return 0
else
local new_nr_of_conns = nr_of_conns -1
local e = {} --element
e[ip] = new_nr_of_conns
kv.insert(tab, e)
return new_nr_of_conns
end
end
The last missing piece is the auth_on_subscribe function that is mainly use to distinguish subscriptions made from 'normal' clients from the ones belonging to 'superusers'. We want to do this as we don't like to expose the $SYS topic tree (useful for monitoring purposes) to everyone. We also want to disallow subscribing to every published message via the root wildcard #, even if this would probably be fun to watch...
Other than that, we allow pretty much everything. The callback is implemented like this:
function auth_on_subscribe(sub)
local logline = "SUBSCRIBE: client: "..sub.client_id
for k, s in pairs(sub.topics) do
local topic = s[1]
local qos = s[2]
if (string.sub(topic, 1, 4) == "$SYS") or (topic == "#") then
-- check whether we are superuser
if not superusers[sub.username] then
-- we don't allow
logline = logline .. " topic: " .. topic .. " qos: " .. qos .."\n"
log.warning("NOT-ALLOWED-" .. logline)
return false
end
end
logline = logline .. " topic: " .. topic .. " qos: " .. qos .."\n"
end
log.info(logline)
return true
end
That's it, we've now covered most of the Lua script that is deployed using vmq-diversity on our public.vernemq.com broker instance. I hope you were able to follow us through those few lines of Lua code. And if not, please don't hesitate to get in touch.
Give it a try and head over to the docs.
Some limitations we should mention at this point:
vmq-diversity uses an Erlang based implementation of Lua that can be found here, which provides a subset of the official Lua distribution. While most of the pure Lua modules should work, the ones relying on C libraries e.g. luasockets won't work here. Another limitation is performance, as a recognisable overhead exists when VerneMQ has to call into a Lua plugin.
Thanks for reading!
Your VerneMQ Mission Engineers!