EMQX Performance Tuning: Max Connections and File Descriptors
Table of Contents
While testing the performance of EMQX, if the number of client connections reaches a large number, you may find that EMQX cannot accept more connections, and the following error log appears in the console or
[error] Accept error on 0.0.0.0:1883: EMFILE (Too many open files)
This indicates that the number of file descriptors opened by EMQX has reached the maximum limit. This is a resource limit enforced by Linux. If we want to lift this limit, we need to modify the corresponding kernel parameters of Linux.
In this article, we will introduce how to modify kernel parameters to increase the number of file descriptors that EMQX can use on Linux.
What is a file descriptor?
A file descriptor is an index created by the Linux system kernel for efficient management of open files. When we open a file using the following function call, fd is the file descriptor returned by the kernel:
fd = open("example.txt", O_RDWR)
Then, we can perform various read and write operations on the opened file through this
fd. Linux will index the correct file according to
In Linux systems, all resources can be treated as files. In addition to normal files such as text files and executable files, file directories, hardware devices, and network sockets (Network Socket) are also considered as files.
As an MQTT server, EMQX will create a socket for each newly connected client. Correspondingly, this will consume a file descriptor.
To prevent the process from inadvertently or maliciously opening too many file descriptors, the Linux system limits the number of file descriptors each user and process can open. The default limit is often hard to meet the demands of an MQTT server with many device connections. Therefore, we have to modify kernel parameters for EMQX.
Tune the maximum limit of file descriptors
In Linux systems, the maximum number of file descriptors that a process can ultimately use is influenced by several kernel parameters or configuration files. They are:
The value defined in
/proc/sys/fs/file-max determines the maximum number of file descriptors available to the entire system. We can view the current limit by using the following command:
If the current value is smaller than our desired value, we can run the following command to modify it:
echo <Number> > /proc/sys/fs/file-max
sysctl -w fs.file-max = <Number>
Both of the above commands can make the changes take effect immediately, but the disadvantage is that the changes are only effective in the current session. Once the user logs out or the system is restarted, our changes will become invalid. If we want to make the changes permanent, we can open the file
/etc/sysctl.conf and add the following content:
fs.file-max = <Number>
Execute the following command to make the changes in
/etc/sysctl.conf take effect immediately:
Generally, we recommend setting the maximum number of file descriptors slightly larger than the target value to avoid affecting the normal operation of other services in the system.
The value defined in
/proc/sys/fs/nr_open determines the maximum setting for the number of file descriptors that each process can open. It does not directly affect the process but determines the maximum value other parameters can be set to.
It is similar to
/proc/sys/fs/file-max, view the current value:
echo <Number> > /proc/sys/fs/nr_open
sysctl -w fs.nr_open = <Number>
Permanent changes can be made by adding the following line to the file
fs.nr_open = <Number>
Then execute the following command to make the changes take effect immediately:
In this file, we can set limits for various resources for users who log in through PAM (Pluggable Authentication Modules), such as the maximum number of processes, and the maximum number of open file descriptors. When we use the
emqx command with the option
start to start EMQX, the maximum number of file descriptors that EMQX can open is limited by
Each line in
limits.conf uses the following format to describe a limit for the user.
<domain> <type> <item> <value>
<domain> determines which users this limit applies to. We can use a username, or group name (preceded by @ to distinguish it from usernames), or directly use the wildcard * to limit all users.
<type> determines whether this is a soft or hard limit. The soft limit is the current actual resource limit, while the hard limit is the upper limit to which the soft limit can be set. After the limits in
limits.conf take effect, only the root user can increase their hard limit. Common users can only adjust their soft limit within the range of the hard limit or decrease the hard limit.
<item> is used to specify the type of resource. For example, nproc represents the maximum number of processes, nofile represents the maximum number of open file descriptors. The available
<item> list can be viewed in
/etc/security/limits.conf. In addition, we should also note the difference between different resource limits. For example, nproc is used to limit the maximum number of processes that a user can create simultaneously, but nofile is not the maximum number of file descriptors that a user can open simultaneously. It's an independent limit for each process created by the user.
<value> is the specific limit size.
Here is a simple example. It indicates that the maximum number of file descriptors that
emqx users can open is 65535 by default, but the
root user can break this limit at any time, the
emqx user cannot. The maximum number of file descriptors that all other users can open is 1024 by default, but can be adjusted up to 4096:
root soft nofile 65535
root hard nofile 65535
emqx soft nofile 65535
emqx hard nofile 65535
* soft nofile 1024
* hard nofile 4096
The soft and hard limits of nofile cannot exceed
/proc/sys/fs/nr_open, which is something to pay special attention to. This is because Linux will not check and limit our modifications, but if we exceed this limit, it will cause the system to be unable to log in.
After we have made changes to
/etc/security/limits.conf, we must log out of the session and log back in again for the changes to take effect. After logging back in, we can use the following command to confirm whether the changes have been successfully applied:
We can also use the
-Hn options to view the currently effective soft and hard limits separately:
Changes in the file
/etc/security/limits.conf are permanent, but sometimes we may want to make temporary changes. In this case, we can use the
ulimit command. We just introduced how to use it to view the current resource limits, we will see how to use it to modify resource limits, for example, modifying both soft and hard limits simultaneously:
ulimit -n <Number>
Or modify the soft and hard limits separately:
ulimit -Sn <Number>
ulimit -Hn <Number>
Please note that we can never the soft limit exceed the hard limit, and non-root users can only reduce the hard limit, only the
root user can freely increase the hard limit. However, no matter what, the soft and hard limits modified by
ulimit should not exceed the value defined in
The changes that the
ulimit command makes to resource limits are temporary, and they only apply to the current Shell and the processes it creates. For example, when we run the following command to create a sub-shell and modify the maximum number of file descriptors in it, this change will not take effect in the parent Shell:
$ (ulimit -n 2000; ulimit -n)
$ ulimit -n
Or when we use the
su command to switch to another user, the system will create a new Shell, which will also cause our changes to become ineffective. In the following example, we list the current Shell's PID to prove that this is a new Shell:
$ echo "The current shell's PID is $$"
The current shell's PID is 2083
$ ulimit -n
$ su - emqx
$ echo "The current shell's PID is $$"
The current shell's PID is 7837
$ ulimit -n
This is why sometimes we find that the changes made by
ulimit do not take effect.
Taking EMQX as an example, if we installed it through a software package or package management tool, EMQX would automatically create an
emqx user during installation and switch to this user at startup. This means that changes we made in the current Shell by running the
ulimit command as a non-emqx user will be ineffective when EMQX is started with
emqx start and switched the user. In this case, the resource limits of EMQX are determined directly by
For these reasons, we generally recommend modifying
/etc/security/limits.conf can only set resource limits for users logging in through PAM, it does not affect the resource limits of system services.
So when we start the EMQX service with
sudo systemctl start emqx or
sudo service emqx start, this service will not be affected by the limitations in
The maximum number of file descriptors available to system services is determined by the parameter
DefaultLimitNOFILE in the file
/etc/systemd/system.conf or the parameter
LimitNOFILE in the file
/usr/lib/systemd/system/<Service Name>.service. The former is a global default setting applied to all system services, while the latter is an optional individual limit for each service.
LimitNOFILE will override
/etc/systemd/system.conf are global and always require a server reboot to take effect. Changes in each service's own unit file can take effect by simply running the command
sudo systemctl daemon-reload to reload the systemd configuration file. So the latter is our recommended configuration method.
In terms of configuration format,
LimitNOFILE and DefaultLimitNOFILE are the same. We can set both soft and hard limits at the same time.
DefaultLimitNOFILE have the same configuration format, using
LimitNOFILE as an example, we can set both hard and soft limits:
LimitNOFILE = <Number>
We can also set soft and hard limits separately:
LimitNOFILE = <Soft Limit>:<Hard Limit>
The currently effective
DefaultLimitNOFILE can be viewed with the command
systemctl show --property DefaultLimitNOFILE, and
LimitNOFILE can be viewed with the command
systemctl show emqx -p LimitNOFILE. However, both of them can only view hard limits, not soft limits.
LimitNOFILE to 1048576 by default in the unit file
/usr/lib/systemd/system/emqx.service. If we have modified it, we can run the following command to make the changes take effect:
systemctl restart emqx
Note that the LimitNOFILE limit should also not exceed the value defined in
/proc/sys/fs/nr_open, or the service will fail to start.
Suppose now we want EMQX to be able to accept 500,000 client connections, we can configure it like this:
/etc/sysctl.conf, and add the following two lines:
fs.file-max = 520000 fs.nr_open = 510000
sysctl -pto make the changes take effect.
/etc/security/limits.conf, and add the following two lines:
emqx soft nofile 510000 emqx hard nofile 510000
Log out of the session and log back in to make the changes take effect.
Open the EMQX unit file
/usr/lib/systemd/system/emqx.serviceand modify the parameter
LimitNOFILE = 510000
systemctl daemon-reloadto make the changes take effect.
Start EMQX with
sudo systemctl emqx start. After successful startup, use the command
ps aux | grep emqxto get the PID of the EMQX process. Then, we can check the effective resource limits of the current process with the following command:
$ cat /proc/<PID>/limits Limit Soft Limit Hard Limit Units ... Max open files 510000 510000 files ...
The maximum connection limit of EMQX
After correctly modifying the above system parameters, the number of connections to EMQX has finally reached our expectations. However, as long as the number of clients continues to increase, it is still possible to exceed our modified limits.
Since the basic services of EMQX will also request file descriptors, if the client connections have already occupied all available file descriptors, the operating system will return an EMFILE error (The per-process limit of open file descriptors has been reached) or an ENFILE error (The system limit on the total number of open files has been reached). This will cause EMQX to be unable to continue to operate normally.
Therefore, we recommend enabling the maximum connection limit of EMQX, with EMQX actively restricting the number of connections to ensure that the number of open file descriptors does not exceed the system's resource limit. This restriction mechanism is disabled by default, we need to enable it by ourselves.
Taking EMQX 5.3.0 as an example, we can access the listener page of the Dashboard, then click on the listener name to enter the corresponding configuration page, and change
Max Connections to a value less than the current system limit:
In Linux systems, there are quite a few parameters that affect the maximum number of file descriptors we can ultimately use. We can only modify the correct parameters according to different situations after understanding their respective uses and scopes. The following diagram can help us further clarify and memorize their relationship:
In subsequent blogs, we will continue to bring more optimization guides for kernel parameters that affect the performance of EMQX on Linux systems.