Confluent/Kafka integration, high-performance bi-direction data transport of massive IoT data
Kafka is a highly scalable, reliable, secure, real-time event streaming platform combining messaging, storage, and data processing. The built-in rule engine and data bridge of EMQX can stream IoT data (MQTT messages) to Kafka. The massive-scale device connectivity provided by #EMQX and the high-throughput, durable data processing capability from #Kafka, make the perfect IoT data infrastructure.
Background
MQTT Benefits
MQTT is specially designed for IoT devices with low performance, low power consumption and unreliable network connection. The protocol is light and flexible enough to meet the needs of fast analysis and the response of devices and diversity of IoT services.
Limitations of Kafka
Kafka is not built for edge IoT communication. Kafka client needs a stable network connection and more hardware resources. At the same time, it lacks the technical features necessary for IoT scenarios, such as keepalives and will messages.
MQTT + Kafka
Kafka focuses on data storage and reading, while EMQX focuses on the communication between the client and the server. EMQX quickly receives and processes messages from a large number of IoT devices. Kafka collects and stores these data and sends them to data analysts for analysis and processing.
Capabilities
Built-in implementation of data bridging with Confluent/Kafka
With the help of EMQX’s data bridge or Rule Action features, it can consume from and/or produce message batches to Kafka. With UI configuration interface, you can easily setup the data pipelines to pipe data between your IoT devices and enterprise data platforms without having to implement it yourself.
Higher performance and automatic capacity expansion
Thanks to the asynchronous and batch processing mechanism, in our stress tests, EMQX is able to write up to 1 million messages per second to Kafka. And EMQX is able to automatically discover the partition scale-ups in Kafka, without having to re-configure or restart, which may cause interruptions to the data stream, making it easy for you to scale up the capacity as your business grows flexibly.
Seamless integration with the rule engine
Through the built-in rule engine of EMQX Enterprise, you can extract, filter, enrich and transform IoT data in real-time, and then write the processed results to a specific Kafka topic. Combined with Kafka's streaming processing capability, you can quickly build application integration and business Innovation.
Features List
- Bidirectional. EMQX supports batching MQTT messages towards Kafka, also fetching Kafka messages to publish to MQTT clients.
- Flexible MQTT-to-Kafka topic mapping, e.g. one-to-one, one-to-many, many-to-many, including MQTT topic filters (wildcards).
- EMQX comes with a local on-disk buffer mechanism to prevent data loss during network disturbances, Kafka service maintenance, or even node restarts of EMQX itself.
- EMQX Kafka producer supports synchronous/asynchronous write mode, making it flexible when it comes to prioritizing latency vs. throughput.
- Dynamically configure data bridging in Dashboard or configuration file.
- Realtime metrics, such as the total number of messages, the number of succeeded/failed deliveries, and messaging rate, etc.
- Integrated with SQL IoT rules to extract, filter, enrich and transform data before pushing the messages to Kafka or devices.