When we started developing VerneMQ we had just finished a large industrial messaging research project. The project focused on centralizing and standardizing the communication between all participants of a liberalized energy marketplace where the participants ranged all the way from power plant operators and the involved suppliers to the energy consumers.
As is the case with most research projects a final prototype should validate the findings of the research. That's the point where Erl.io, the company behind VerneMQ, jumped in to help out with the implementation of the prototype. The group had already decided to base their prototype on the AMQP broker RabbitMQ. Since the research group didn't have the required know-how we, as RabbitMQ experts, where given the chance to work on the prototype.
One of our main tasks was to teach the group of university researchers, industrial project managers, and engineers from local SMEs about the possibilities of RabbitMQ. To make a long story short, we learned how a message broker should be architected and implemented to fit large scale messaging projects. The key requirements are:
Scalability: Large scale messaging projects with hundreds of thousands to millions of participants require a scalable core infrastructure, that can leverage multicores as well as different styles of horizontal scaling strategies, such as cloud, commodity hardware, containers, and virtualized infrastructures. Moreover, a cost efficient path to mitigate load peaks must be part of the solution.
Reliability: The larger a messaging infrastructure grows the more money could potentially be lost if the service experiences abnormal downtimes. Therefore a message broker must be carefully engineered with the philosophy A System will fail, be prepared in mind.
Security: Every system that faces millions of clients must protect itself against faulty miss-behaving clients as much as possible, e.g. via rate limits, message size limitation, fine granularity ACLs, disabling unfavorable protocol features to name a few mechanisms.
Extensibility: One of the big challenges large scale messaging projects have is that they are required to integrate with many different business systems, e.g. existing BI solutions, SQL databases, workflow engines, AAA systems etc. A messaging middleware must be a broker as well as a framework and therefore cannot make too many assumptions about the way users want to integrate it.
Operations: A large scale messaging system is ideally started exactly once in its system life cycle. This requires that maintenance and upgrades can be done without stopping the system. The more clients and servers are involved in such a system the more complex operations and maintenance become. For this reason it's important that the system works with and for the operators and not against them.
During our work on the mentioned prototype we learned that neither RabbitMQ nor any other middleware that currently exists implements all those key requirements, which results in large investments for patching or fixing the existing solutions.
Moreover we've also discovered that heavy weight protocols such as AMQP (0.9.1 to be specific) and XMPP are not the good protocols when a huge number of devices will connect to the broker. And since we didn't want to reinvent the wheel and design our own messaging protocol we started looking for lightweight pub/sub alternatives suitable for IoT scenarios. Although not flawless the MQTT protocol has proven to be the perfect IoT protocol.
This was the start for VerneMQ!