Open MQTT Benchmark Suite: The Ultimate Guide to MQTT Performance Testing

Introduction

We are thrilled to announce that the Open MQTT Benchmark Suite presented by EMQ is now available! With typical and practical use cases, primary metrics for measuring broker performance, and a tool to simulate loads and collect benchmark results, our Open MQTT Benchmark Suite can help you evaluate the scalability and performance of MQTT brokers to make an informed selection.

Born for Impartial MQTT Load Testing

The MQTT broker is central in connecting devices and transferring IoT data in the publish/subscribe model context. With IoT proliferating, the challenging problem of handling massive devices and high data throughput makes an MQTT broker's scalability and performance crucial for any IoT solution.

The Open MQTT Benchmark Suite is designed to provide an impartial and practical way to test MQTT brokers. It will simplify MQTT broker load testing with the following main advantages:

Offering an open and practical evaluation base

Despite numerous research papers and technical writeups evaluating and comparing MQTT brokers, the current benchmarks typically measure only specific factors and do not reflect the real-world scenarios of large-scale IoT applications. Our Open MQTT Benchmark Suite thoroughly examines the key factors for evaluation from various aspects and designs scenarios that are impartial towards any brokers, with the goal of establishing an unbiased testing base.

Based on real use cases and feasible metrics for measurement

We have analyzed and addressed a significant number of actual testing requirements from our customers, and based on that, we have developed this Open MQTT Benchmark Suite. We are confident that this benchmark suite can fulfill the majority of testing requirements.

Open to the community for a comprehensive benchmark

We are committed to building an open and collaborative community around our MQTT benchmark suite. By including diverse use cases contributed by the community, our suite will accurately reflect the needs and requirements of the industry. Join us in building a comprehensive benchmark suite for the MQTT protocol and help us empower users to be well-informed about their MQTT broker choices.

What’s Inside the Open MQTT Benchmark Suite

The Open MQTT Benchmark Suite's first edition introduces the key factors affecting MQTT broker performance and defines metrics for measuring scalability, availability, latency, and computing resource cost.

Next, the benchmark suite categorizes typical usage scenarios into connection, fan-out, point-to-point, and fan-in and includes practical use cases based on real-world customer requirements.

To illustrate how these use cases are applied in benchmarking, we provide several examples with detailed benchmark results.

An Example Benchmark Result

To demonstrate a practical example, we present below the detail of a fan-out use case and benchmarking result using NanoMQ as the MQTT broker.

In this fan-out use case, connections for 5 publishers and 1,000 subscribers are established (i.e., more subscribers than publishers for “fan-out" purposes), and 5 topics are used for Publish/Subscribe. Once a subscriber is connected, it immediately subscribes to all 5 topics. Later on, each publisher sends a 16-byte message to an exclusive topic. Both publishers and subscribers use QoS 1. The publish rate for each publisher is 50 messages per second, and consequently, the total expected subscription rate is 250K messages per second.

To perform benchmarking, NanoMQ is deployed on a single node. The configuration details are:

XMeter is used as the tool to perform benchmarking. It simulates all clients and messages throughput, analyzing data and rendering metrics reports. The highlighted metrics are:

The detailed charts are also appended:

Future Works

We use NanoMQ as an example, but the benchmark suite targets all brokers implementing the MQTT protocol and will continuously iterate and evolve. We are also going to propose a tool for the MQTT benchmarking.

If you are interested in this MQTT benchmark suite, we highly recommend you check out the GitHub and get more information. And any contributions to make the benchmark suite better are welcome!