Test Purpose

In EMQ X 3.0, we have done the benchmark test of 10 million connections and 1 million message throughput. In that test, a cluster of 20 EMQ X nodes was used, and the machine of each node had 16 cores & 32G.

In version 4.0 and subsequent 4.2 and 4.3, we redesigned and continuously optimized the architecture and internal modules, which greatly improved the performance of EMQ X. In order to demonstrate the capabilities of EMQ X 4.3 in large-scale MQTT concurrent connections, topic subscriptions, and high message throughput in this test, the tested cluster has only 5 nodes, and the machine resources of each node are 32 cores and 64G.

Test Scenarios

Huawei cloud host is used in this test, and 5 EMQ X nodes and the test machines are in the same vpc. The main scenarios are 10 million MQTT connections, 10 million topic subscriptions,1 million QoS0 message throughput, and 500,000 QoS1 message throughput, as shown below.

Note: Unless otherwise specified, all connections have a heartbeat time of 300 seconds by default, and all message payloads are 50 bytes.

Scenario 1: 10 million MQTT connections and subscriptions + QoS0 broadcast scenario with 1 million message throughput

10 million MQTT clients connect to the EMQ X cluster at a rate of 20,000 new connections per second. Each client subscribes to a topic after successful connection, and every 10 connections subscribe to the same topic. Therefore, the test reaches 1 million topics and 10 million subscriptions. After 10 million connections and subscriptions are completed, the message broadcast scenario will start (the number of receiving terminals is much larger than that of publishing terminals). 50 MQTT connections are used as pub clients to publish messages. Every 10 connections publish messages to the topic testn/${machineName()}/ ${threadNum} (n is 1~5) as a group. Each pub client publishes 100 QoS 0 messages per second. 1000 sub clients are also divided into 5 groups, each with 200 subscription topics of testn / # (n is 1 ~ 5).

Therefore, the total message publishing throughput rate is 5000 messages per second, and the total message receiving throughput rate reaches 1 million per second.

Scenario 2: 10 million MQTT connections and subscriptions + QoS0 one-to-one with 1 million message throughput

MQTT 10 million connections and subscriptions are the same as the above scenario. After 10 million connections and subscriptions are completed, one-to-one messaging will start (the number of receiving terminals is equal to the publishing terminals, and each receiving terminal subscribes to a corresponding pub topic). Both pub clients and sub clients are 25,000. Each pub client publishes 20 QoS 0 messages per second, and each sub client consumes 20 QoS 0 messages. Therefore, both publishing and receiving are 500,000 messages per second, and the total message throughput reaches 1 million.

Scenario 3: 10 million MQTT connections and subscriptions + QoS1 broadcast scenario with 500,000 message throughput

MQTT 10 million connections and subscriptions are the same as in scenario 1 above. After 10 million connections and subscriptions are completed, the message broadcast scenario with QoS of 1 starts (the number of receiving terminals is much larger than that of publishing terminals). 50 MQTT connections are used as pub clients to publish messages, and every 10 publish messages to the topic testn/${machineName ()}/${threadNum} (n is 1~5) as a group. Each pub client sends 50 QoS 1 messages per second. The 1000 sub clients are also divided into 5 groups, each with 200 subscription topics testn/# (N is 1 to 5).

Therefore, the total message publishing throughput rate is 2,500 messages per second, and the total message receiving throughput rate reaches 500,000 per second.

Scenario 4: 10 million MQTT connections and subscriptions + QoS1 one-to-one with 500,000 message throughput

MQTT 10 million connections and subscriptions are the same as in scenario 1 above. After 10 million connections and subscriptions are completed, one-to-one messaging with QoS of 1 will start (the number of receiving terminals is equal to the publishing terminals, and each receiving terminal subscribes to a corresponding pub topic). Both pub clients and sub clients are 25,000. Each pub client publishes 10 QoS 1 messages per second, and each sub client consumes 10 QoS 1 messages. Therefore, both publishing and receiving are 250,000 messages per second, and the total message throughput reaches 500,000.

Test Result Highlights

Scenario 1: 10 million MQTT connections and subscriptions + QoS0 broadcast scenario with 1 million message throughput

  • The average connection speed reaches 20,000 per second, and the average response time for connection + subscription is 4.2ms
  • Sub average response time is 0.33ms
  • The average CPU usage of EMQ X on each node is 34%. After 10 million clients are all connected, the CPU usage range of EMQ X on each node during messaging period is 32%~60%.
  • The average memory usage of EMQ X on each node is 28.4GB. After 10 million clients are all connected, the memory usage range of EMQ X on each node is 29GB~35.4 GB during messaging period.
Min. response time (s) Max. response time (s) Average response time (s) Average throughput EMQ X CPU usage EMQ X memory usage(G)
MQTT Connection 0.002 0.03 0.0042 19978 32%~60% 29~35.4
MQTT Pub / / / 5002.462
MQTT Sub 0 0.178 0.0003 1000490

Scenario 2: 10 million MQTT connections and subscriptions + QoS0 one-to-one with 1 million message throughput

  • The average connection speed reaches 20,000 per second, and the average response time for connection + subscription is 5ms
  • Sub average response time is 0.3ms
  • The average CPU usage of EMQ X on each node is 51%. After 10 million clients are all connected, the CPU usage range of EMQ X on each node during messaging period is 56%~68%.
  • The average memory usage of EMQ X on each node is 28.7GB. After 10 million clients are all connected, the memory usage range of EMQ X on each node is 28.6GB~35.2GB during messaging period.
Min. response time (s) Max. response time (s) Average response time (s) Average throughput EMQ X CPU usage EMQ X memory usage(G)
MQTT Connection 0.002 3.076 0.005 19923 56%~68% 28.6~35.2
MQTT Pub / / / 505045
MQTT Sub 0 0.04 0.0003 502322

Scenario 3: 10 million MQTT connections and subscriptions + QoS1 broadcast scenario with 500,000 message throughput

  • The average connection speed reaches 20,000 per second, and the average response time for connection + subscription is 4.6ms
  • Pub average response time is 1ms, and Sub average response time is 0.3ms
  • The average CPU usage of EMQ X on each node is 51%. After 10 million clients are all connected, the CPU usage range of EMQ X on each node during messaging period is 35%~57%.
  • The average memory usage of EMQ X on each node is 28.5GB. After 10 million clients are all connected, the memory usage range of EMQ X on each node is 29.6GB~33.8GB during messaging period.
Min. response time (s) Max. response time (s) Average response time (s) Average throughput EMQ X CPU usage EMQ X memory usage(G)
MQTT Connection 0.002 0.37 0.0046 19998 35%~57% 29.6~33.8
MQTT Pub 0 0.123 0.001 2607
MQTT Sub 0 0.116 0.0003 521555

Scenario 4: 10 million MQTT connections and subscriptions + QoS1 one-to-one with 500,000 message throughput

  • The average connection speed reaches 20,000 per second, and the average response time for connection + subscription is 4.5ms
  • Pub average response time is 1.1ms, and Sub average response time is 0.6ms
  • The average CPU usage of EMQ X on each node is 47%. After 10 million clients are all connected, the CPU usage range of EMQ X on each node during messaging period is 50%~65%.
  • The average memory usage of EMQ X on each node is 28.8GB. After 10 million clients are all connected, the memory usage range of EMQ X on each node is 30.9GB~36.4GB during messaging period.
Min. response time (s) Max. response time (s) Average response time (s) Average throughput EMQ X CPU usage EMQ X memory usage(G)
MQTT Connection 0.002 3.022 0.0045 19803 50%~65% 30.9~36.4
MQTT Pub 0 0.283 0.0011 249192
MQTT Sub 0 0.03 0.0006 521555

Test Tools

The enterprise version of the XMeter performance test platform is used in this test.

XMeter overview (https://www.xmeter.net/): XMeter is a performance test platform based on the extension of the open source test tool JMeter. In view of the large-scale access, flexible expansion requirements, multiple access protocols, and mixed scenarios of the Internet of Things, XMeter has modified JMeter to support large-scale and high-concurrency performance test, such as tens of millions of MQTT concurrent connection testing. In addition to testing the MQTT protocol, it can also support mainstream application test, such as HTTP / HTTPS.

JMeter-MQTT Plug-in: It is an open source MQTT performance test plug-in implemented by XMeter, which has been used in many projects and is currently the most popular MQTT plug-in in the JMeter community.

Tool version:

  • XMeter:Enterprise 3.0
  • JMeter-MQTT Plug-in:mqtt-xmeter-2.0.2
  • JMeter:JMeter5.2.1

Get the Report

The download URL will be sent to the email you filled in.