Add business context
Combine device events with maintenance, production, or business records in Databricks to investigate operational questions.
EMQX Databricks Integration
Bring MQTT data into Databricks
Prepare MQTT telemetry with EMQX, store it in Amazon S3, and bring it into Databricks for SQL analytics and machine learning.
Overview
Put device data to work in Databricks
Combine telemetry with business data to investigate operations and build datasets for analytics and AI.
Store data you can use
Filter messages and shape payloads in EMQX before writing files, so your lakehouse receives the fields your analytics need.
Separate ingestion from analytics
Use S3 as a shared storage layer. EMQX uploads device data while Databricks reads and processes files on its own schedule.
Architecture
From MQTT messages to Databricks queries
EMQX prepares messages and writes files to Amazon S3. Databricks reads them through a Unity Catalog external location for SQL queries and further processing.
- Collect
MQTT devices
Telemetry and events
- Prepare & route EMQX
Rules + S3 Sink
Filter · transform · upload
- Land
Amazon S3
Your bucket and prefix
- Analyze
Databricks
External location · SQL · AI
From signals to datasets
| Device | Temp °C | Observed |
|---|---|---|
| sensor-01 | 23.8 | 09:42:18 |
| sensor-02 | 24.1 | 09:42:21 |
| sensor-03 | 22.6 | 09:42:24 |
Capabilities
Control what reaches your lakehouse
Filter MQTT messages, choose file formats, and organize S3 uploads for Databricks.
Filter and transform
Select topics and payload fields with SQL rules. Filter out unwanted messages and transform records before the S3 sink writes them.
Choose formats and batching
Write individual messages or batch records into CSV, JSON Lines, or Parquet files. Set record-count or time limits for each batch, using formats supported by your EMQX version.
Organize files in S3
Set S3 object keys with supported template variables to group files by time or upload sequence, giving Databricks a consistent path to read.
How it works
Set up ingestion in four steps
Use an EMQX deployment with S3 integration, an S3 bucket, and a Databricks on AWS workspace with Unity Catalog.
- 01
Configure S3 access
Choose a bucket and prefix. Create a Databricks storage credential and an external location for that S3 path.
- 02
Connect EMQX to S3
Create an Amazon S3 connector with your region’s endpoint and AWS credentials, then test the connection.
- 03
Define the rule and sink
Create a SQL rule, attach an S3 sink, and set the object key, file format, and upload method.
- 04
Publish and verify
Publish a test MQTT message. Check the resulting file in S3, then read it from Databricks to confirm the data and format.
Use cases
Turn telemetry into operational insight
Analyze equipment history, compare sites, and prepare data for machine learning in Databricks.
Equipment trend analysis
Compare temperature, vibration, and status history with maintenance records to spot changes in equipment behavior.
Explore industrial IoTCompare performance across sites
Combine telemetry with production, energy, and location data to compare performance across sites, assets, and time periods.
Datasets for machine learning
Build historical telemetry datasets for feature engineering, model training, and evaluation in Databricks.
Resources
Dive deeper into EMQX & Databricks
EMQX documentation
Databricks integration guide
Set up the S3 workflow and verify delivery from MQTT publishing to a Databricks query.
EMQX documentation
Amazon S3 integration
Configure S3 uploads, object keys, batching, and file formats for your telemetry.
EMQX community
Databricks integration with Zerobus
Follow the EMQX team’s updates on the native Zerobus connector targeted at EMQX 7.0.
FAQ
Frequently asked questions
The workflow shown here uses Amazon S3. EMQX prepares MQTT messages and writes files through its S3 sink. Databricks reads those files through a Unity Catalog external location.
Direct uploads store the content you configure. Aggregated uploads support CSV, JSON Lines, and Parquet, depending on your EMQX version. Choose a format your Databricks workflow can read and test it with representative messages.
Freshness depends on upload batching, S3 delivery, and when Databricks reads or processes the files. For aggregated uploads, tune the record-count and time limits to balance file size and data freshness.
EMQX needs permission to write to the selected S3 bucket and prefix. Databricks needs an IAM-backed storage credential, an external location, and the Unity Catalog privileges required to read those files.
Yes. EMQX SQL rules can match topics, select payload fields, and filter or transform messages before the S3 sink writes the files Databricks reads.
Build your Databricks ingestion pipeline
Plan your MQTT ingestion, S3 storage, and Databricks workflows with an EMQX expert.