Save MQTT Data from EMQX Cloud on GCP to InfluxDB Cloud through the public network
EMQX Cloud Team
Table of Contents
In this article, we will simulate temperature and humidity data and report it to the EMQX cloud on the GCP platform via the MQTT protocol, after which we will use the EMQX Cloud data integration to enable the NAT gateway and save the data to the InfluxDB Cloud over the public network.
EMQX Cloud
EMQX Cloud is the world's first fully managed MQTT 5.0 public cloud service for IoT from EMQ. It provides a one-stop O&M colocation and a unique isolated environment for MQTT services. In the era of Internet of Everything, EMQX Cloud can help you quickly build industry applications and easily realize the collection, transmission, computation, and persistence of IoT data.
With the infrastructure provided by cloud providers, EMQX Cloud is available in dozens of countries and regions around the world, providing low-cost, secure, and reliable cloud services for 5G and Internet of Everything applications.
For more information, please go to the EMQX Cloud website or see the EMQX Cloud documentation.
InfluxDB Cloud
InfluxDB Cloud is a fully managed and hosted version of InfluxDB 2.0, the time series platform purpose-built to collect, store, process and visualize metrics and events. As an open source chronological database developed by InfluxData, focusing on high performance reading and writing; efficient storage and real-time analysis of massive chronological data, etc. It is ranked No.1 in the DB-Engines Ranking chronological database, and is widely used in DevOps monitoring, IoT monitoring, real-time analysis, and other scenarios.
InfluxDB is simple to deploy and easy to use, taking full advantage of the Go language in its technical implementation, and can be deployed independently without any external dependencies. Provides a SQL-like query language with a friendly and easy-to-use interface. Rich aggregation and sampling capabilities, flexible Retention Policy to set the retention time and number of copies of data, while ensuring data reliability; timely deletion of expired data to free up storage space, and flexible continuous query to achieve the sampling of massive data. A wide range of protocols is supported. In addition to native protocols such as HTTP and UDP, it is also compatible with the communication protocols of components such as CollectD, Graphite, OpenTSDB, and Prometheus.
Create Deployments
Create EMQX Cluster
Create a GCP deployment of EMQX Cloud, other options default.
When the status is Running, the creation of the deployment is complete.
Create InfluxDB Cloud instances
If you are creating an InfluxDB Cloud instance for the first time, you can refer to the help document. Go to the Load Data page and find a new token. You could choose to activate/deactivate the token.
After logging in to InfluxDB's console, go to the Load Data page and create a new bucket. Name the bucket and click Create.
Enable NAT Gateway
NAT gateways can provide network address translation services to provide Professional deployments with the ability to access public network resources without the need for VPC peering connections.
Data Integrations
1. Create InfluxDB HTTP V2 resources
Go to the Data Integrations page. On the data integration page, click InfluxDB HTTP V2 Service resources.
Fill in the InfluxDB Cloud connection details, and then click test. Please check the InfluxDB service if the test fails. Click the New button after the test is passed and you will see the Create Resource successfully message.
2. Create a new rule
After the resource is successfully created, you can return to the data integration page and find the newly created resource, and click create rule. Our goal is that as long as the emqx/test topic has monitoring information, the engine will be triggered. Certain SQL processing is required here:
- Only target the topic "temp_hum/emqx"
- Get the three data we need: location, temperature, humidity
According to the above principles, the SQL we finally get should be as follows:
SELECT
payload.location as location,
payload.temp as temp,
payload.hum as hum
FROM "temp_hum/emqx"
You can click SQL Test under the SQL input box to fill in the data:
- topic: temp_hum/emqx
- payload:
{
"temp": 26.3,
"hum": 46.4,
"location":"Prague"
}
Click Test to view the obtained data results. If the settings are correct, the test output box should get the complete JSON data as follows:
If the test fails, please check whether the SQL is compliant and whether the topic in the test is consistent with the SQL filled in.
3. Add Action to Rule
After completing the rule configuration, click Next to configure and create an action. Then enter the fields and tags as follows:
Measurement: temp_hum
Fields: temp ${temp}, hum ${hum}
Tags: location ${location}
Verification
1. Use MQTTX to simulate temperature and humidity data reporting
We recommend you to use MQTTX, an elegant cross-platform MQTT 5.0 desktop client to subscribe/publish messages.
Click on the add button and fill in the deployment information to connect to the deployment. You need to replace broker.emqx.io with the created deployment connection address, add client authentication information to the EMQX Cloud console. Enter the topic name and payload message to publish the message.
2. View rules monitoring
Check the rule monitoring and add one to the number of success.
3. View results in InfluxDB console
Go back to the InfluxDB console and go to the Data Explorer page. Select the bucket and filter the measurement, fields, then InfluxDB will generate the tables and graphs for you.
So far, we have used EMQX Cloud data integration based on the GCP platform to save the entire process of data to the InfluxDB cloud over the public network.
Related resources
- Integrating MQTT Data into InfluxDB for a Time-Series IoT Application
- MQTT Performance Benchmark Testing: EMQX-InfluxDB Integration
- Getting Data from EMQX Cloud with InfluxDB Native Collector
- Supercharging IIoT with MQTT, Edge Intelligence, and InfluxDB