Overview
Distributed measurement systems are essential for modern environmental science, enabling the continuous collection of atmospheric data across vast, often hard-to-access locations. At the Leibniz Institute for Tropospheric Research (TROPOS), the Department of Atmospheric Microphysics (AMP) is leveraging these systems to redefine indoor air quality research.
As a key partner in the EU-funded EDIAQI project, TROPOS deployed custom-developed sensors in private households to monitor indoor pollutants. To manage the resulting data streams, TROPOS implemented EMQX as their central MQTT platform, ensuring scientific-grade data reliability and seamless remote operations.
The Challenge: Reliable Data at Scale
Operating a distributed sensor network within research infrastructures like ACTRIS (Aerosol, Clouds and Trace Gases Research Infrastructure) presents unique technical hurdles:
- Geographic Distribution: Sensors are placed in private homes and remote areas where physical on-site maintenance is difficult and intrusive.
- System Heterogeneity: Managing a diverse landscape of devices requires a unified communication standard.
- Data Consistency: Long-term scientific analysis relies on unbroken time series. Any connection instability leads to data gaps that complicate cleaning and interpretation.
- Future-Proofing: The architecture must support decades of operation, scaling as data volumes and real-time requirements grow.
The central question was: how to connect distributed sensors to collect continuous, synchronized data with reasonable operational effort and a maintainable architecture.
The Solution: A Resilient, Multi-Layered MQTT Architecture
To address these challenges, TROPOS designed a robust architecture centered on EMQX, chosen for its superior clustering capabilities and seamless container-based deployment. The solution prioritizes stability, redundancy, and spatial decoupling:
1. Centralized Coordination with High Availability
The backbone of the system is an EMQX cluster consisting of two nodes positioned behind a load balancer. This setup ensures 24/7 continuous operation, allowing the system to remain functional even if individual components fail.
2. Geo-Distributed Redundancy via Bridging
Beyond the central cluster, TROPOS integrated additional EMQX deployments at separate physical locations using Bridge Mode. This configuration enables cross-site data synchronization and provides critical redundancy, which is a vital requirement for long-term measurements in distributed research infrastructures.
3. Lightweight & Non-Intrusive Data Flow
The custom-developed AQBIE (Air Quality Beacon & Immission Evaluator) sensors utilize MQTT as a lightweight, event-based protocol. This allows for stable, real-time data transmission from many distributed devices without the need for permanent physical access, ensuring unobtrusive operation in private environments.
4. Integrated Monitoring and Fault Detection
The architecture transforms raw data into actionable insights through a integrated stack:
- Storage: Data is streamed into TimescaleDB for long-term time-series analysis.
- Visibility: Grafana dashboards and Prometheus provide real-time system monitoring. This allows the team to identify anomalies and sensor deviations centrally before they impact the final scientific datasets.
The Result: From Data Collection to a Sustainable Platform
By adopting MQTT through EMQX, the TROPOS team has moved beyond simple data collection to a professionalized operational platform. Even in the pilot phase, the benefits are clear:
- Stable & Synchronized Datasets: The MQTT-based infrastructure ensures continuous real-time data collection, providing a more reliable basis for scientific analysis of ventilation behavior and air quality.
- Early Fault Classification: Sensor states and data transmission are now transparently traceable. Deviations are identified and classified during collection, preventing them from appearing as inexplicable gaps or inconsistencies during later analysis stages.
- Minimized Manual Overhead: The effort required for maintenance, fault analysis, and subsequent data cleaning has been drastically reduced.
- Reduced Operational Efforts: Thanks to the container-based deployment, the system remains manageable for scientific teams without deep IT specialization, even as operational complexity grows.
- Scalable Future Options: The architecture now supports more fine-grained real-time monitoring and the easy integration of additional sensors or new projects without a disproportionate increase in complexity.
Looking Ahead
The success of the EDIAQI project has established MQTT and EMQX as integral components of the TROPOS data strategy. As part of the ACTRIS European research infrastructure, designed for a lifespan of over 10 years, this standardized architecture provides a robust foundation for high-quality atmospheric data collection and inter-institutional collaboration.
For more details: MQTT Architecture for Distributed Environmental Sensors