Save MQTT Data from EMQX Cloud to AWS DynamoDB through the public network

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. DynamoDB lets you offload the administrative burdens of operating and scaling a distributed database so that you don't have to worry about hardware provisioning, setup and configuration, replication, software patching, or cluster scaling. DynamoDB also offers encryption at rest, which eliminates the operational burden and complexity involved in protecting sensitive data.

In this article, we will simulate temperature and humidity data and report it to the EMQX Cloud via the MQTT protocol, after which we will use the EMQX Cloud data integration to enable NAT gateway and save the data to the AWS DynamoDB over the public network.

Introduction to EMQX Cloud

EMQX Cloud is the world's first fully managed MQTT 5.0 public cloud service for IoT from EMQ. EMQX Cloud provides a one-stop O&M colocation and a unique isolated environment for MQTT services. In the era of Internet of Everything, EMQX Cloud can help you quickly build industry applications and easily realize the collection, transmission, computation and persistence of IoT data.

With the infrastructure provided by cloud providers, EMQX Cloud is available in dozens of countries and regions around the world, providing low-cost, secure, and reliable cloud services for 5G and Internet of Everything applications.

For more information, please go to the EMQX Cloud website or see the EMQX Cloud documentation.

Create Deployments

Create EMQX Cluster

Once logged in, click "Cloud Console" under the account menu and you will be able to see the green button to create a new deployment. EMQX Cloud offers a 14-day free trial of Standard and Professional plans. This tutorial uses the professional deployment as a demonstration.

Create an AWS deployment of EMQX Cloud, other options default.

When the status is Running, the creation of the deployment is complete.

Create DynamoDB instances

If you are creating a DynamoDB instance for the first time, you can refer to the help document. First, go to the DynamoDB console and click Create Table.

Fill in key information such as table name, partition key, and other defaults, which can be set according to your actual business needs.

Until the status of the table is active, it means that you have successfully created the table 'temp_hum'.

Enable NAT Gateway

NAT gateways can provide network address translation services to provide Professional deployments with the ability to access public network resources without the need for VPC peering connections.

Data Integrations

1. Create DynamoDB resources

   Go to the Data Integrations page. On the data integration page, click DynamoDB resources.

   

   Fill in the DynamoDB connection details, and then click test. Please check the DynamoDB service if the test fails. Click the New button after the test is passed and you will see the Create Resource successfully message.

   

2. Create a new rule

   After the resource is successfully created, you can return to the data integration page and find the newly created resource, and click create rule. Our goal is that as long as the temp_hum/emqx topic has monitoring information, the engine will be triggered. Certain SQL processing is required here:

   • Only target the topic "temp_hum/emqx"
   • Get the three data we need: temperature, humidity

   According to the above principles, the SQL we finally get should be as follows:

   SELECT 
   id as msgid,
   topic, 
   payload 
   
   FROM "temp_hum/emqx"
   

   

   You can click SQL Test under the SQL input box to fill in the data:

   • topic: temp_hum/emqx
   • payload:

   {
     "temp": 26.3,
     "hum": 46.4
   }
   

   Click Test to view the obtained data results. If the settings are correct, the test output box should get the complete JSON data as follows:

   

   If the test fails, please check whether the SQL is compliant and whether the topic in the test is consistent with the SQL filled in.

3. Add Action to Rule

   After completing the rule configuration, click Next to configure and create an action. Then enter the fields and tags as follows:

    DynamoDB Table：temp_hum
    Hash Key：msgid
   

   

Verification

1. Use MQTTX to simulate data reporting

   We recommend you to use MQTTX, an elegant cross-platform MQTT 5.0 desktop client to subscribe/publish messages.

   Click on the add button and fill in the deployment information to connect to the deployment. You need to replace broker.emqx.io with the created deployment connection address, add client authentication information to the EMQX Cloud console. Enter the topic name and payload message to publish the message.

   

2. View rules monitoring

   Check the rule monitoring and add one to the number of success.

   

3. View results in NoSQL Workbench

   NoSQL Workbench for Amazon DynamoDB is a cross-platform client-side GUI application for modern database development and operations. You can reach out to it to connect to DynamoDB.

   

   Go to the Operation Builder page. Select the table 'temp_hum'. Here you can see the results of the temperature and humidity data forwarding.

   

Summary

So far, we have used EMQX Cloud data integration to save the entire process of data to the AWS DynamoDB over the public network. Then, you can integrate with AWS services to do more with your data, like using built-in tools to perform analytics, extract insights, and monitor traffic trends, which will save you more time to focus on building your IoT applications.