Deploying EMQX 4.0 cluster on Kubernetes via Helm3

Helm3 was released in November 2019, which adds many new features compared to Helm2.In this article, how to deploy EMQX cluster on Kubernetes via Helm3 will be introduced.

New features of Helm3

  • Remove Tiller
  • Different namespaces can use the same Release Name
  • Simplify template object .Capabilities
  • Use JSONSchema to validate the values of charts
  • Merge requirements.yaml into Chart.yaml
  • require to specify a Release Name when you install helm. require to use the --generate-name parameter to enable automatic generation.
  • Support pushing to remote registry (eg: harbor)
  • Remove helm serve
  • Command line changes (retain the original command as alias Aliases)
    • helm delete --> helm uninstall
    • helm inspect -> helm show
    • helm fetch -> helm pull
  • go Import path changes k8s.io/helm --> helm.sh/helm

For specific new features, you can refer to the Helm official document

Install Helm3

We provide official scripts of Helm3 to simplify the installation steps, you can execute curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3|bash for one-click installation, or view the Helm official documentation

  • Add helm repository

    $ helm repo add emqx https://repos.emqx.io/charts
    $ helm repo update
    
  • Check EMQX

    helm search repo emqx
    NAME           CHART VERSION  APP VERSION DESCRIPTION
    emqx/emqx      v4.0.0         v4.0.0      A Helm chart for EMQX
    emqx/emqx-ee v4.0.0           v4.0.0      A Helm chart for EMQX
    emqx/kuiper    0.1.1          0.1.1       A lightweight IoT edge analytic software
    
  • Start EMQX cluster and setservice.type=NodePort

    $ helm install my-emqx emqx/emqx --set service.type=NodePort
    
  • Check the cluster status of EMQX

    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1     Running   0          56s
    my-emqx-1  1/1     Running   0          40s
    my-emqx-2  1/1     Running   0          21s
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: #{running_nodes =>
                          ['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local',
                           'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',
                           'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local'],
                      stopped_nodes => []}
    
  • Check EMQX service

    $ kubectl get svc
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                      AGE
    my-emqx              NodePort       10.101.143.92    <none>        1883:32756/TCP,8883:31569/TCP,8081:30585/TCP,8083:31804/TCP,8084:30523/TCP,18083:31253/TCP   4m33s
    my-emqx-headless     ClusterIP      None             <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP                                       4m33s
    

It can be seen that the host IP corresponding to port 18083 of my emqx is 31253. (NodePort changes during each deployment, which is subject to the actual deployment.)

  • Access port 31253 of any Kubernetes node IP, enter the default username: admin, default password: public, and log in to the EMQX dashboard.

  • Delete EMQX cluster.

    $ helm uninstall my-emqx
    release "my-emqx" uninstalled
    

Deploy a persistent EMQX cluster

EMQX persists pods by creating a PVC resource mount/opt/emqx/data/mnesia directory. Before deploying EMQX, users need to deploy a load balancer such as Haproxy or Nginx-PLUS and Creating PVC resources or Storage Classes resources in Kubernetes.

  • Start EMQX cluster

    • If the user deployed a PVC resource, set persistence.existingClaim=your_pv_name
    $ helm install my-emqx emqx/emqx --set persistence.enabled=true --set persistence.existingClaim=your_pv_name
    
    • If the user deployed a Storage Classes resource, setpersistence.storageClass=your_storageClass_name

      $ helm install my-emqx emqx/emqx --set persistence.enabled=true --set persistence.storageClass=your_storageClass_name
      
  • Check EMQX cluster status

    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1     Running   0          56s
    my-emqx-1  1/1     Running   0          40s
    my-emqx-2  1/1     Running   0          21s
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: #{running_nodes =>
                          ['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local',
                           'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',
                           'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local'],
                      stopped_nodes => []}
    
  • Taking Storage Classes as an example, you can see that the PVC resource has been successfully established

    $ kubectl get pvc
    NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    emqx-data-my-emqx-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            2m11s
    emqx-data-my-emqx-1   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            99s
    emqx-data-my-emqx-2   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            56s
    

    The cluster will mount the /opt/emqx/data/mnesia directory of EMQX to the PVC. After the Pods are rescheduled, EMQX will obtain data from the /opt/emqx/data/mnesia directory and restore it.

  • Check ClusterIP of EMQX

    $ kubectl get svc
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                  AGE
    my-emqx              ClusterIP   10.100.205.13   <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP   26m
    my-emqx-headless     ClusterIP   None            <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP   26m
    

You can see that the ClusterIP of my-emqx is10.100.205.13 (ClusterIP will change during each deployment, which is subject to the actual deployment)

  • Forward port 1883, 8883, 8081, 8083, 8084, 18083 of the URL monitored by the load balancer to the clusterIP of my-emqx. If there is a need for TLS connection, it is recommended to terminate the SSL connection at the load balancer. The connection between client and load balancer is a TLS secure connection, and the connection between LB and EMQX is an ordinary TCP connection.

  • Access URL:18083, enter the default username: admin, default password: public, and log in to EMQX dashboard.

  • Use the helm upgrade command to easily expand the EMQX cluster. The following example shows the helm upgrade command by adding an EMQX node

    # Changed the number of nodes in EMQX to 5.
    # Note: The number of nodes in EMQX is recommended to be odd
    $ helm upgrade --set replicaCount=5 my-emqx emqx/emqx
    Release "my-emqx" has been upgraded. Happy Helming!
    
  $ kubectl get pods
  NAME       READY  STATUS             RESTARTS  AGE
  my-emqx-0  1/1    Running            0         4m25s
  my-emqx-1  1/1    Running            0         4m14s
  my-emqx-2  1/1    Running            0         4m
  my-emqx-3  1/1    Running            0         31s
  my-emqx-4  1/1    Running            0         15s

  $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
  Cluster status: #{running_nodes =>
                        ['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local',
                         'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',
                         'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local',
                         'my-emqx@my-emqx-3.my-emqx-headless.default.svc.cluster.local',
                         'my-emqx@my-emqx-4.my-emqx-headless.default.svc.cluster.local'],
                    stopped_nodes => []}
  • Delete EMQX cluster

    $ helm uninstall my-emqx
    release "my-emqx" uninstalled
    

Note: After the EMQX cluster is deleted, the PVC resources will not be released automatically than EMQX can be restored. You need to manually delete the PVC resources after confirming that you do not need to restore EMQX.

  $ kubectl get pvc
  NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  emqx-data-my-emqx-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            84m
  emqx-data-my-emqx-1   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            84m
  emqx-data-my-emqx-2   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            83m
  emqx-data-my-emqx-3   Bound     pvc-b6c5a565-adbd-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            25m
  emqx-data-my-emqx-4   Bound     pvc-c626cafd-adbd-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            25m

  $ kubectl delete pvc emqx-data-my-emqx-0 emqx-data-my-emqx-1 emqx-data-my-emqx-2 emqx-data-my-emqx-3 emqx-data-my-emqx-4                    
  persistentvolumeclaim "emqx-data-my-emqx-0" deleted
  persistentvolumeclaim "emqx-data-my-emqx-1" deleted
  persistentvolumeclaim "emqx-data-my-emqx-2" deleted
  persistentvolumeclaim "emqx-data-my-emqx-3" deleted
  persistentvolumeclaim "emqx-data-my-emqx-4" deleted

Deploy EMQX Edge cluster and EMQX cluster of Enterprise Edition

EMQX Edge

Specify image.repository=emqx/emqx-edge when deploying EMQX Edge cluster, and other settings are consistent with the deployment of EMQX cluster

$ helm install my-emqx-edge emqx/emqx --set image.repository=emqx/emqx
$ kubectl get pods
NAME            READY   STATUS    RESTARTS   AGE
my-emqx-edge-0  1/1     Running   0          35s
my-emqx-edge-1  1/1     Running   0          23s
my-emqx-edge-2  1/1     Running   0          9s

EMQX EE

When deploying an cluster of EMQX Enterprise Edition, you need to log in to emqx.io to apply and download a License file at first, and create the License file as a Secret resource

$ kubectl create secret generic your-license-secret-name --from-file=/path/to/emqx.lic

Then, specify the repo as emqx/emqx-ee,emqxLicneseSecretName=your-license-secret-name during deployment, and other settings are consistent with the deployment of EMQX cluster

$ helm install my-emqx-ee emqx/emqx-ee --set emqxLicneseSecretName=your-license-secret-name

EMQX Helm Chart Configuration Item

Parameter Description Default Value
replicaCount For the number of EMQX nodes, it is recommended to keep an odd number , otherwise it will not be able to recover automatically after brain splitting 3
image.repository EMQX Image name emqx/emqx
image.pullPolicy Pulling Image Policy IfNotPresent
persistence.enabled Whether to enable PVC false
persistence.storageClass Storage class Name nil
persistence.existingClaim PV Name ""
persistence.accessMode PVC access mode ReadWriteOnce
persistence.size PVC size 20Mi
resources CPU/ memory resource {}
nodeSelector pod assigned node labels {}
tolerations []
affinity {}
service.type Emqx cluster service type ClusterIP
emqxConfig EMQX Configuration item,see documentation for details {}
`emqxLicneseSecretName For EMQX Enterprise Edition, it needs to manually create a Secret resource from a license file as (only valid in emqx / emqx-e) ""

When you need to set complex parameters, you can use Yaml files to record the parameters

$ helm install my-emqx emqx/emqx -f values.yaml

You can get default values.yaml from Github .

Try EMQX Cloud for Free
A fully managed, cloud-native MQTT service
Get Started →

Related Posts

EMQX 4.4 new feature: a top view of slowest subscribers

As a MQTT broker admin, you may often want to know which clients are the slowest. We have a nice ‘top’ view for you.

2022-03-14
The Development Work for EMQX 5.0 is in Full Swing - EMQX Newsletter 202107

EMQX development work is in full swing this July. We have added some exciting features and made a breakthrough progress on the key tasks of version 5.0.

2021-08-05
Build an EMQX cluster based on HAProxy

HAProxy can provide high availability, load balancing, and TCP and HTTP based application proxies. This article will introduce how to build the EMQX cluster based on HAProxy.

Zhiwei Yu 2021-01-27