WHITE PAPER
Driving the Future of Connected Cars with MQTT →

Open MQTT Benchmarking Comparison: Mosquitto vs NanoMQ

May Jin
Apr 21, 2023
Open MQTT Benchmarking Comparison: Mosquitto vs NanoMQ

The blog post Open MQTT Benchmark Suite: The Ultimate Guide to MQTT Performance Testing introduced the Open MQTT Benchmark Suite developed by EMQ. We defined MQTT benchmark scenarios, use cases, and observation metrics in the GitHub project. Based on the activity and popularity of the community and GitHub project, the top 4 open-source MQTT brokers in 2023 – EMQX, Mosquitto, NanoMQ, and Vernemq, were chosen to perform the benchmark test.

This blog series presents the benchmark test results and aims to help you choose a suitable MQTT broker based on your needs and use cases.

In this first post, we provide the benchmarking results of NanoMQ and Mosquitto brokers. Additionally, we compare the features and capabilities of both brokers in detail in another post.

MQTT Benchmark Scenarios and Use Cases

The MQTT Benchmark Suite designs two sets of benchmark use cases. One is named Basic Set, which is for small-scale performance verification, and another is called Enterprise Set, which aims for enterprise level verification.

Detailed descriptions of the testing scenarios are already available on the GitHub project, but for convenience, we briefly list them here as well.

All the tests are executed on a single node.

Use Cases

Basic Set

  • Point-to-Point: p2p-1K-1K-1K-1K
    • 1k publishers, 1k subscribers, 1k topics
    • Each publisher pubs 1 message per second
    • QoS 1, payload 16B
  • Fan-out: fanout-1-1k-1-1K
    • 1 publisher, 1 topic, 1000 subscribers
    • 1 publisher pubs 1 message per second
    • QoS 1, payload 16B
  • Fan-in: sharedsub-1K-5-1K-1K
    • 1k publishers, 1k pub topics
    • 5 subscribers consume all messages in a shared subscription way
    • Publish rate: 1k/s (each publisher pubs a message per second)
    • Shared subscription’s topic: $share/perf/test/#
    • Publish topics: test/$clientid
    • QoS 1, payload 16B
  • Concurrent connections: conn-tcp-10k-100
    • 10k connections
    • Connection rate (cps): 100/s

Enterprise Set

  • Point-to-Point: p2p-50K-50K-50K-50K
    • 50k publishers, 50k subscribers, 50k topics
    • Each publisher pubs 1 message per second
    • QoS 1, payload 16B
  • Fan-out: fanout-5-1000-5-250K
    • 5 publishers, 5 topics, 1000 subscribers (each sub to all topics)
    • Publish rate: 250/s, so sub rate = 250*1000 = 250k/s
    • QoS 1, payload 16B
  • Fan-in: sharedsub-50K-500-50K-50K
    • 50k publishers, 50k pub topics
    • Publish rate: 50k/s (each publisher pubs a message per second)
    • Use a shared subscription to consume data (to avoid slow consumption by subscribers affecting broker performance, 500 subscribers are used to share the subscription)
    • Shared subscription’s topic: $share/perf/test/#
    • Publish topics: test/$clientid
    • QoS 1, payload 16B
  • Concurrent connections: conn-tcp-1M-5K
    • 1M connections
    • Connection rate (cps): 5000/s

Common MQTT Config

Config Value
keep alive 300s
clean session true
authentication enablement no
TLS authentication enablement no
test duration 30 minutes

Testbed

The test environment is configured on AWS, and all virtual machines are within a VPC (virtual private cloud) subnet.

Broker Machine Details

  • Public cloud: AWS
  • Instance type: c5.4xlarge 16C32G
  • OS: Ubuntu 22.04.1 amd64

Test Tool

XMeter is used in this benchmark test to simulate various business scenarios. XMeter is built on top of JMeter but with enhanced scalability and more capabilities. It provides comprehensive and real-time test reports during the test. Additionally, its built-in monitoring tools are used to track the resource usage of the Mosquitto/NanoMQ server, enabling a comparison with the information provided by the operating systems.

XMeter provides a private deployment version (on-premise) and a public cloud SaaS version. A private XMeter is deployed in the same VPC as the MQTT broker server in this testing.

XMeter Architecture

SW Version

Broker Version
Mosquitto (with persistence disabled) 2.0.15
NanoMQ 0.17.0
XMeter 3.2.4

Benchmarking Results

Basic Set

point-to-point: 1K:1K

Average pub-to-sub latency (ms) Max CPU user+system Avg CPU user+system Max memory used Avg memory used
Mosquitto 0.25 0% 0% 278M 254M
NanoMQ 0.25 1% 0% 271M 270M

Fan-out 1k QoS 1

Average pub-to-sub latency (ms) Max CPU user+system Avg CPU user+system Max memory used Avg memory used
Mosquitto 5.73 0% 0% 270M 260M
NanoMQ 13.66 0% 0% 271M 263M

Fan-in 1k - shared subscription QoS 1

Average pub-to-sub latency (ms) Max CPU user+system Avg CPU user+system Max memory used Avg memory used
Mosquitto 0.20 0% 0% 281M 246M
NanoMQ 0.18 0% 0% 294M 267M

10K connections cps 100

Average latency (ms) Max CPU user+system Avg CPU user+system Max memory used Memory used Stable at
Mosquitto 0.6 0% 0% 306M 264M
NanoMQ 0.59 0% 0% 320M 320M

Enterprise Set

point-to-point: p2p-50K-50K-50K-50K

Metrics

Actual msg rate Average pub-to-sub latency (ms) Max CPU user+system Avg CPU user+system Max memory used Avg memory used
Mosquitto 37k:37k 353.82 6% 6% 341M 318M
NanoMQ 50k:50k 91 35% 30% 1.33G 1.3G

In this scenario, Mosquitto cannot reach to the target message rate. It stabilized at 37300/s for both pub and sub.

NanoMQ keeps the stable pub & sub rate at 50000/s during the 30-minutes' test.

pub-to-sub latency percentiles

pub-to-sub latency percentiles

Latency (ms) Mosquitto NanoMQ
p50 361 82
p75 367 171
p90 372 210
p95 378 225
p99 417 251

Result Charts

  • Mosquitto

    Mosquitto Result Charts

  • NanoMQ

    NanoMQ Result Charts

Fan-out: fanout-5-1000-5-250K

Metrics

Actual msg rate Average pub-to-sub latency (ms) Max CPU user+system Avg CPU user+system Max memory used Avg memory used
Mosquitto 82k 12,240.83 7% 6% 355M 341M
NanoMQ 255k 13.91 73% 71% 781M 682M

In this scenario, Mosquitto cannot reach to the target message rate. The throughput has been fluctuating around 80,000/s.

NanoMQ keeps the stable rate at 250,000+/s throughout the test.

pub-to-sub latency percentiles

pub-to-sub latency percentiles

Latency (ms) Mosquitto NanoMQ
p50 12,378 14
p75 12,522 18
p90 12,571 21
p95 12,596 23
p99 12,627 26

Result Charts

  • Mosquitto

    Mosquitto Result Charts

  • NanoMQ

    NanoMQ Result Charts

Fan-in: sharedsub-50K-500-50K-50K

Metrics

Actual msg rate Average pub-to-sub latency (ms) Max CPU user+system Avg CPU user+system Max memory used Avg memory used
Mosquitto pub: 50ksub: 40k 12,723.07 7% 7% 485M 456M
NanoMQ pub: 50ksub: 50k 2.76 34% 34% 795M 783M

In this scenario, the consumption rate of Mosquitto cannot reach to the target rate. It stabilized at 41,000/s.

NanoMQ keeps the stable pub & sub rate at 50,000/s throughout the test.

pub-to-sub latency percentiles

pub-to-sub latency percentiles

Latency (ms) Mosquitto NanoMQ
p50 13,138 2
p75 13,281 3
p90 13,423 4
p95 13,526 5
p99 13,736 21

Result Charts

  • Mosquitto

    Mosquitto Result Charts

  • NanoMQ

    NanoMQ Result Charts

Concurrent Connection: conn-tcp-1M-5K

Metrics

Average latency (ms) Max CPU user+system Avg CPU user+system Max memory used Memory used Stable at
Mosquitto 5.74 2% 2% 1G 1G
NanoMQ 3.16 5% 4% 6.9G 6.9G

Latency percentiles

Latency percentiles

Latency (ms) Mosquitto NanoMQ
p50 2 2
p75 2 2
p90 2 2
p95 2 2
p99 9 3

Result Charts

  • Mosquitto

    Mosquitto Result Charts

  • NanoMQ

    NanoMQ Result Charts

Conclusion

As mentioned in the blog post, Mosquitto is a single-threaded design, while NanoMQ is built on NNG's asynchronous I/O with a built-in actor multi-threading model. The above test results have proven it. Under the basic set, the performance of these two is similar, while in the enterprise level tests, NanoMQ fully utilizes the multi-core and multi-threading capabilities to support higher and more stable message throughput.

For example, in the fan-out scenario, Mosquitto can only support an outgoing message rate of around 82k per second, while NanoMQ can keep a stable rate of 250k. Even NanoMQ can stably handle message throughput of up to 500k on this c5.4xlarge virtual machine.

While Mosquitto struggles to contain the workload, NanoMQ can easily fit into the boots. However, it doesn't mean NanoMQ beats Mosquitto, and only they are designed for different targets: single-core vs. multi-cores. It is becoming common for embedded designs to incorporate more than one CPU on an SMP platform. That said, NanoMQ is the best fit if you work with a multi-core platform and want to achieve the best throughput. On the other hand, Mosquitto can help you limit CPU and memory usage while performance is not a priority.

Try NanoMQ for Free
Get Started →

Related Posts