WHITE PAPER
Driving the Future of Connected Cars with MQTT →

Ingesting IoT Data From EMQX Enterprise to GCP Pub/Sub

Guowei Li
Apr 17, 2023
Ingesting IoT Data From EMQX Enterprise to GCP Pub/Sub

You may have already successfully connected your IoT devices to EMQX Enterprise using the instructions provided in our previous posts. This article will introduce how to ingest IoT data from EMQX Enterprise to GCP Pub/Sub service to help you complete the migration.

Create and Manage Pub/Sub Topics

If Pub/Sub has not been enabled yet, you can create a topic by following these steps.

  1. Open the Pub/Sub console, click CREATE TOPIC, input a custom Topic ID, and click CREATE to create it.

    Topics

    Create Topic

  2. Navigate to the topic details page by clicking on the corresponding Topic ID from the list. Once on the page, create a subscription to store messages. Choose the Pull type and specify a message retention duration of 7 days.

    Please refer to GCP Pub/Sub subscription for comprehensive information regarding subscriptions.

    Create a subscription 1

    Create a subscription 2

  3. Click Subscription IDMESSAGESPULL to view the messages sent to the topic.

    Click Subscription ID

    Click PULL to view the messages

After successfully creating a Pub/Sub topic and gaining familiarity with its basic management operations, we can now use EMQX Enterprise to forward data to the newly created topic.

Forward MQTT Messages to Pub/Sub

EMQX Enterprise offers GCP Pub/Sub integration for easy IoT Core migration. Even if you are not an IoT Core user, you can still forward client events and messages to the relevant Pub/Sub topics, integrating IoT data with applications and services on Google Cloud.

The following steps are to configure Pub/Sub integration on EMQX Enterprise.

Create a Service Account JSON

The Service Account JSON is a file containing authentication and authorization credentials, and EMQX Enterprise must access Pub/Sub resources.

  1. Go to the GCP Console, input IAM in the search box, and go to the IAM & Admin page.

    Search IAM & Admin

  2. Navigate to the IAM & Admin page and click on Service AccountsEmail. Then select the appropriate email address and click on the KEYS tab. From there, click ADD KEY to create a new key in JSON format for authentication. Be sure to keep this file secure.

    Click Service Accounts

Create a Pub/Sub Resource

  1. Open the EMQX Dashboard, go to Rule EngineResources, click Create, and select GCP PubSub on the pop-up box.

    EMQX Dashboard Rule Engine

  2. Provide a Resource ID to identify the resource and paste the previously generated Service Account JSON into it. Before creating the resource, it is recommended to click on Test to ensure connectivity. If no issues are found during the connectivity test, click Confirm to complete the creation process.

    Create Resource

  3. Once the resource is successfully created, it will appear as available. The next step is to configure the rule for forwarding data to this resource.

    Resource list

Create a Rule to Forward Data to Pub/Sub

  1. Click RuleCreate to enter the rule creation page. In the SQL field, input the following content.

    SELECT
      *
    FROM
      "t/#"
    

    The SQL statement will be executed when a message is received that matches the t/# topic. The SELECT clause will then retrieve the client context and message payload, which will be used in the action.

    For more SQL functions and use cases, please refer to the EMQX Rule Engine.

    Create Rules

  2. Click Add action at the bottom of the page, select the action type Data forward, choose Data to GCP Pubsub, and select the resource just created to forward the SQL output to Pub/Sub.

    Add action 1

  3. Input the pre-created Topic ID into the GCP PubSub Topic field, and construct the message that needs to be forwarded in the Payload template. By default, the payload template is empty and will forward the entire SQL output result. Once you have made the necessary changes, click on the Confirm button to complete the creation of the action.

    Add action 2

  4. After confirming everything is correct, click Create to complete rule creation.

    Click Create to complete rule creation

Testing and Verification

In the previous steps, a rule was created to forward all messages that match the t/# topic to the my-iot-core topic in Pub/Sub.

Next, we will use the WebSocket tool to perform a verification test.

  1. Go to the ToolWebSocket page, establish a client connection, and publish several messages to the t/1 topic.

    Tool → WebSocket page

  2. On the Rule page, click the Monitor icon in the table to view the statistics of the specified rule. You can see that data_to_gcp_pubsub has 4 records, indicating that the data has been successfully written to Pub/Sub.

    Click the Monitor icon

  3. On the details page of the Pub/Sub subscription, you can view the corresponding message received by the topic.

    Details page of the Pub/Sub subscription

Summary

We have covered the steps and procedures to migrate IoT Core services to an EMQX Enterprise cluster within the user's GCP environment.

To learn more about what you can do with EMQX Enterprise, please visit EMQX Enterprise Documentations or contact us to book a demo.

Try EMQX Enterprise for Free
Connect any device, at any scale, anywhere.
Get Started →

Related Posts