Master MQTT from zero to expert

Sometimes we also directly refer to the server as Broker, and the two terms can be used interchangeably.

The client can use this field during the connection to indicate whether it expects to resume communication from an existing session or create a completely new session. This is only applicable for MQTT v5.0.

A device or application that connects to a server using the MQTT protocol, and they publish and subscribe through the server.

The Client ID is used to uniquely identify a client’s connection and session. MQTT allows the client to specify the Client ID by itself, and also supports the uniform assigning of the Client ID to the client by the server.

The network connection between an MQTT client and an MQTT broker. There is no direct connection between MQTT clients.

Used to describe the content type of the message for easier handling by the receiver. MIME types such as text/plain can be used, as well as custom strings to describe the message content. Only applicable in MQTT v5.0.

MQTT v5.0 introduced the AUTH packet to support enhanced authentication. It extends the password-based and token-based authentication provided by the Username and Password fields.

It is more like an authentication framework that allows for the use of various more secure authentication mechanisms. For example, SCRAM authentication supports mutual confirmation of identity between the server and client to resist man-in-the-middle attacks.

MQTT v5.0 introduces a flow control mechanism, which allows the client and the server to agree on the maximum message sending rate of the other party according to their own receiving capabilities, avoiding problems such as network congestion and receiver overload caused by one party sending too quickly.

Keep Alive indicates the maximum interval that is allowed to elapse between the transmission of MQTT control packets from the client.

If no other control packets can be sent, the client must send a PINGREQ packet.

If the server does not receive any control packets from the client within 1.5 times the Keep Alive interval, it will disconnect.

This typically refers to PUBLISH packets.

MQTT v5.0 allows clients to set expiration time for messages to avoid messages that have been stored in the server for a long time being forwarded to subscribers.

"MQTT over..." refers to what protocol MQTT runs on top of. Common examples include MQTT over TCP, MQTT over TLS, and so on.

The MQTT protocol only requires the underlying transport layer to provide ordered, reliable bidirectional byte streams and does not mandate the use of a specific transport protocol.

MQTT over QUIC is an extension of MQTT by EMQX. Compared to TCP, QUIC can effectively reduce connection overhead and message latency in weak network environments. It also has features such as multiplexing, connection migration, and end-to-end encryption, providing a more flexible and reliable transport layer for modern mobile internet.

The MQTT specification released by the OASIS Technical Committee in October 2014.

The MQTT specification released by the OASIS Technical Committee in March 2019 is also the latest MQTT specification. While introducing many new features, MQTT v5.0 remains backward compatible with v3.1.1.

This typically refers to MQTT control packets. The MQTT protocol communicates by exchanging pre-defined MQTT control packets.

Examples of MQTT control packets include the CONNECT and AUTH (only available in MQTT v5.0) packets used for connection, the PUBLISH packet used for publishing, and the SUBSCRIBE packet used for subscribing, among others.

The packet identifier is used to uniquely identify a message with QoS greater than 0 or a subscribe/unsubscribe request between the client and the server.

The setting of the packet identifier is usually done inside the client and the server.

The payload part in the MQTT packet, depending on the packet type, the content of the payload will be different.

For a PUBLISH packet, the payload is the actual content of the message. For a SUBSCRIBE packet, the payload refers to the subscription list.

However, in most cases, unless otherwise specified, the term “Payload“ refers to the actual content of the message in the PUBLISH packet.

Used to indicate whether the message content (including the Will message) is a UTF-8 encoded string. Only applicable in MQTT v5.0.

The client needs to send a PINGREQ packet to the server in a timely manner to inform it that it is still alive.

The server also needs to respond with a PINGRESP packet in a timely manner so that the client can determine the network and server's activity status.

MQTT defines a set of optional properties for most control packets. Different types of control packets have different optional properties.

For example, Session Expiry Interval is an optional property of CONNECT packet, and Topic Alias is an optional property of PUBLISH packet.

The publish-subscribe mechanism is the core of the MQTT protocol. It decouples the message sender (publisher) and receiver (subscriber), and introduces the role of an intermediate agent to route and distribute messages.

Publishers and subscribers don't need to know each other's existence, and the only connection between them is a consistent agreement on the message, such as what topic the message will use, what fields the message will contain, and so on.

Through the publish-subscribe mechanism, we can realize the broadcast, multicast and unicast of messages.

MQTT defines three levels of QoS to provide different levels of message reliability guarantee. Each message can independently set its own QoS when published.

QoS 0: At most once delivery, messages may be lost.

QoS 1: At least once delivery, messages can be guaranteed to arrive, but may arrive multiple times.

QoS 2: Exactly once delivery, messages are guaranteed to arrive and will not be duplicated.

MQTT indicates the result of the operation through the Reason Code field in the response packet such as CONNACK.

MQTT v5.0 extends Reason Code to reflect more accurate results, and makes all response packets support Reason Code.

Reason Codes are typically machine-readable, so MQTT also provides Reason String to carry human-readable content, which further indicates the result based on Reason Code. This feature is only available in MQTT v5.0.

It is used to declare the maximum number of QoS 1 and QoS 2 messages that the server and the client are willing to concurrently process, and the peer needs to abide by this limit when sending messages. MQTT v5.0 only.

The publish-subscribe mechanism of MQTT ensures that the publisher can only guarantee that the message has reached the server, but not necessarily the subscriber. Therefore, additional request-response mechanisms are required.

MQTT v5.0 improves support for request-response. The requester can directly specify the response topic in the request, so there is no need for the requester and responder to agree on the topic beforehand. They can also ensure the correct matching of requests and responses by correlation data.

Retained messages are not only forwarded like normal messages, but also retained on the MQTT broker.

When a new subscription is created, if there are any retained messages that match the subscription on the server, these retained messages will be sent to the subscriber.

The server can only store one latest retained message for each topic.

MQTT supports multiple security mechanisms, including but not limited to: using TLS at the transport layer to provide end-to-end secure connections, protecting messages from eavesdropping, tampering or forgery; supporting client and server authentication at the MQTT protocol level, as well as authorization checks on clients, to ensure that only authorized users can access specific topics.

A device or application that acts as an intermediary between publishing clients and subscribing clients, whose primary responsibility is to forward all received messages to matching subscribing clients.

MQTT v5.0 allows the server to send a DISCONNECT packet to indicate to the client the specific reason for the connection being closed.

MQTT v5.0 allows the server to indicate to the client, through the Server Reference property in the CONNACK or DISCONNECT packet, to temporarily or permanently switch to another server.

The session mechanism of MQTT ensures that the protocol flow of QoS 1 and 2 messages can be realized.

A session is a stateful interaction between the client and the server, which stores state information such as the transmission status of QoS 1 and 2 messages and subscription information.

It can last only as long as a network connection, or it can exist across multiple network connections. We usually refer to the latter as persistent sessions.

We can choose to resume the connection from an existing session, or we can choose to start from a brand new session.

It represents how long the session can be kept alive on the server after the client connection is disconnected. This is only applicable for MQTT v5.0.

The server informs the client whether the current connection is a continuation of the previous session or a brand new session through this field, so that the client can make corresponding adjustments.

MQTT v5.0 allows clients to be divided into multiple subscription groups, and messages will still be forwarded to all subscription groups, but clients within the same subscription group will alternate in receiving messages using random, round-robin, or other strategies.

This allows subscribers to consume messages in a load-balanced way.

Some MQTT servers, such as EMQX, provide support for shared subscriptions for non-MQTT v5.0 devices outside of the protocol.

The client can specify a subscription identifier when subscribing, and the server needs to include the associated subscription identifier when forwarding messages that match these subscriptions.

In certain use cases, subscription identifiers can help reduce the amount of data that needs to be transmitted or help the client determine which callback to trigger for a message.

MQTT allows clients to use different subscription options for each of their subscriptions, such as whether to receive retained messages when subscribing, and the maximum QoS level for messages that the server can sends to them.

In MQTT v3.1.1, only setting the maximum QoS is supported.

Topics are used to identify and distinguish different messages, which is the basis of MQTT message routing. Publishers can specify the topic of the message when publishing, and subscribers can choose to subscribe to the topics they are interested in to receive related messages.

MQTT v5.0 allows the sender to map a topic name to an alias represented by a two-byte integer, and then replace the original topic name with the alias during message transmission to reduce bandwidth consumption.

The topic filter is used when subscribing and it can contain topic wildcards to subscribe to multiple topics simultaneously.

The topic name is used when publishing a message and topic wildcards are not allowed.

MQTT provides two topic wildcards: the single-level wildcard represented by "+" and the multi-level wildcard represented by "#". Wildcards can only be used in topic filters.

MQTT provides optional Username and Password fields in the CONNECT packet to support password authentication and token-based authentication.

MQTT v5.0 allows clients and servers to add custom, unlimited string key-value pairs as metadata to all control packets except for PINGREQ and PINGRESP, providing better scalability.

User properties can be passed between the client and the server, or between clients, depending on the specific control packet type.

This is used to indicate how long the server should delay sending the will message after a client disconnects. This is only applicable for MQTT v5.0.

If a client disconnects from the broker unexpectedly, the server will forward the will message set by the client during connection to other clients.

Like normal messages, will messages have fields such as topic, QoS, payload, and the retained message flag, and so on.

Topics starting with "$" must be determined by the server for their usage and scenarios, and clients cannot use such topics arbitrarily for their own purposes.

For example, topics starting with "$share" are used for shared subscriptions, and topics starting with "$SYS" are usually used by the server to publish system messages.

EMQX also defines the "$delay" prefix for implementing delayed message delivery.