EMQX Dedicated New Feature: Event History is available for private beta test. →

Building K8S cluster of EMQX starting from scratch

EMQX Team
Jun 30, 2020
EMQX
Building K8S cluster of EMQX starting from scratch

EMQX Team provides Helm chart which facilitates users to one-click deploy EMQX MQTT broker in the Kubernetes cluster. EMQX Team mostly recommends This method for deploying EMQX MQTT broker in the Kubernetes or k3s cluster. This article will start from scratch using the handwriting YAML file method to deploy a K8S cluster of EMQX MQTT broker, and analyze details and techniques of the deployment. It will facilitate users to flexibly use during real deployment.

Reading this article needs users to know the basic concept of Kubernetes and having an operational Kubernetes cluster.

Deploy single EMQX MQTT broker node in K8S

Use pod directly deploy EMQX Broker

In Kubernetes, the smallest management element is Pod than individual containers. Pod is the basic executing unit of Kubernetes applications, which means that it is the smallest and simplest unit be created or deployed in the Kubernetes object model. Pod represents the processes running on the cluster.

EMQX Broker has provided mirroring in docker hub, so users can easily deploy EMQX Broker in the single pod. Using the command kubectl run to create a pod running EMQX Broker.

$ kubectl run emqx --image=emqx/emqx:v4.1-rc.1  --generator=run-pod/v1
pod/emqx created

View the status of EMQX Broker:

$ kubectl get pods -o wide
NAME   READY   STATUS    RESTARTS   AGE
emqx   1/1     Running   0          3m13s

$ kubectl exec emqx -- emqx_ctl status
Node 'emqx@192.168.77.108' is started
emqx 4.1-rc.1 is running

Delete pod:

$ kubectl delete pods emqx
pod "emqx" deleted

Pod is not designed as a persistence resource, it will not survive in the scheduling failing, node crashing or other recovering(such as because of lack of resource or other maintenances). Therefore, a controller is needed to manage Pod.

Use Deployment deploy Pod

Deployment provides a declarative definition(declarative) method for pod and ReplicaSet to replace the previous ReplicationController for easily managing applications. The typical applying scenarios including:

  • Define Deployment to create Pod and ReplicaSet
  • Scroll upgrade and rollback applications
  • Expand and shrink
  • Pause and continue Deployment

Use Deployment deploy an EMQX Broker Pod：

  • Define Deployment：

    $ cat deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: emqx-deployment
  labels:
    app: emqx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083

  • Deploy Deployment：

    $  kubectl apply -f deployment.yaml
deployment.apps/emqx-deployment created

  • View Deployment situation:

    $ kubectl get deployment
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/emqx-deployment   3/3     3            3           74s

$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
pod/emqx-deployment-7c44dbd68-8j77l   1/1     Running   0          74s

$ kubectl exec pod/emqx-deployment-7c44dbd68-8j77l -- emqx_ctl status
Node 'emqx-deployment-7c44dbd68-8j77l@192.168.77.117' is started
emqx 4.1-rc.1 is running

  • Trying manually delete Pod

    $ kubectl delete pods emqx-deployment-7c44dbd68-8j77l
pod "emqx-deployment-7c44dbd68-8j77l" deleted

$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
emqx-deployment-68fcb4bfd6-2nhh6   1/1     Running   0          59s

    The output result represents that successfully using Deployment deploy EMQX Broker Pod. Even if this pod is accidentally stopped, Deployment will recreate a new pod.

Use services exposing EMQX Broker Pod service

Kubernetes Pods has a life cycle. They can be created, and will not run if they are destroyed. It can dynamically create and destroy pod, if use Deployment to run applications.

Every pod has its IP address, but in Deployment, the pod collection running at the same moment may differ from that runs this application later.

This will cause a problem: if use EMQX Broker Pod to provide service to the MQTT client, how does the client find and track the IP address that will be connected, to facilitate users to use EMQX Broker service.

The answer is: Service

Service is an abstract method for exposing the application which is running on a set of Pods as network service.

Use Service to expose EMQX Broker Pod as network service.

  • Define Service：

    $cat service.yaml

apiVersion: v1
kind: Service
metadata:
  name: emqx-service
spec:
  selector:
    app: emqx
  ports:
    - name: mqtt
      port: 1883
      protocol: TCP
      targetPort: mqtt
    - name: mqttssl
      port: 8883
      protocol: TCP
      targetPort: mqttssl
    - name: mgmt
      port: 8081
      protocol: TCP
      targetPort: mgmt
    - name: ws
      port: 8083
      protocol: TCP
      targetPort: ws
    - name: wss
      port: 8084
      protocol: TCP
      targetPort: wss
    - name: dashboard
      port: 18083
      protocol: TCP
      targetPort: dashboard

  • Deploy Service：

    $ kubectl apply -f service.yaml
service/emqx-service created

  • View deployment situation

    $ kubectl get svc
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                        AGE
emqx-service   ClusterIP   10.96.54.205   <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP   58s

  • Use the IP provided by Service to view API of EMQX Broker

    $ curl 10.96.54.205:8081/status
Node emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120 is started
emqx is running

So far, the deployment of a single EMQX Broker node is done. Manage EMQX Broker Pod through Deployment, and expose EMQX Broker service through Service.

Automatically cluster EMQX MQTT broker through Kubernetes

The article above introduced how to deploy single EMQX Broker Pod through Deployment. It is extremely convenient to expand the number of Pod through Deployment, execute the command kubectl scale deployment ${deployment_name} --replicas ${numer} can expand the number of Pod. Expanding EMQX Broker Pod to three is as follows:

$ kubectl scale deployment emqx-deployment --replicas 3
deployment.apps/emqx-deployment scaled

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
emqx-deployment-68fcb4bfd6-2nhh6   1/1     Running   0          18m
emqx-deployment-68fcb4bfd6-mpvch   1/1     Running   0          6s
emqx-deployment-68fcb4bfd6-mx55q   1/1     Running   0          6s

$ kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 -- emqx_ctl status
Node 'emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120' is started
emqx 4.1-rc.1 is running

$ kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120'],
                  stopped_nodes => []}

You can see that the number of EMQX Broker Pod is expanded to three, but each pod is separated, and there are no clusters. Next try to automatically cluster EMQX Broker Pod through Kubernetes.

Modify EMQX Broker configuration

View the content related to automatically cluster in the EMQX Broker documentation, you can see that we need to modify the configuration of EMQX Broker.

cluster.discovery = kubernetes
cluster.kubernetes.apiserver = http://10.110.111.204:8080
cluster.kubernetes.service_name = ekka
cluster.kubernetes.address_type = ip
cluster.kubernetes.app_name = ekka

cluster.kubernetes.apiserver is the address of Kubernetes apiserver, we can get it through this command kubectl cluster-info. cluster.kubernetes.service_name is the Service name we mentioned above. cluster.kubernetes.app_name is the part before the @ symbol in node.name of EMQX Broker, so you also need to set the EMQX Broker in the cluster as a uniform prefix of node.name.

The docker mirroring of EMQX Broker provides the function of modifying configurations through environment variables. For more details, you can check docker hub or Github.

  • Modify the yaml file of Deployment and add environment variables:

    $ cat deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: emqx-deployment
  labels:
    app: emqx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083
        env:
        - name: EMQX_NAME
          value: emqx
        - name: EMQX_CLUSTER__DISCOVERY
          value: k8s
        - name: EMQX_CLUSTER__K8S__APP_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__SERVICE_NAME
          value: emqx-service
        - name: EMQX_CLUSTER__K8S__APISERVER
          value: "https://kubernetes.default.svc:443"
        - name: EMQX_CLUSTER__K8S__NAMESPACE
          value: default

    Because this command `kubectl scale deployment ${deployment_name} --replicas ${numer} will not modify file yaml, needs to set spec.replicas: 3 when modify yaml.

    https://kubernetes.default.svc:443 will be parsed as the address of Kubernetes apiserver because of the build-in DNS rules of Kubernetes.

  • Delete the previous Deployment and then redeploy:

    $ kubectl delete deployment emqx-deployment
deployment.apps "emqx-deployment" deleted

$ kubectl apply -f deployment.yaml
deployment.apps/emqx-deployment created

Give authority to pod for accessing Kubernetes apiserver

After successfully deploy Deployment, check the status of EMQX Broker, then you can see that although EMQX Broker runs successfully, the cluster is still not successful. Check the log of EMQX Broker Pod:

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
emqx-deployment-5c8cfc4d75-67lmt   1/1     Running   0          5s
emqx-deployment-5c8cfc4d75-r6jgb   1/1     Running   0          5s
emqx-deployment-5c8cfc4d75-wv2hj   1/1     Running   0          5s

$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt -- emqx_ctl status
Node 'emqx@192.168.87.150' is started
emqx 4.1-rc.1 is running

$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt -- emqx_ctl cluster status
Cluster status: #{running_nodes => ['emqx@192.168.87.150'],
                  stopped_nodes => []}

$ kubectl logs emqx-deployment-76f6895c46-4684f

···
(emqx@192.168.87.150)1> 2020-05-20 01:48:39.726 [error] Ekka(AutoCluster): Discovery error: {403,
                                     "{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"endpoints \\\"emqx-service\\\" is forbidden: User \\\"system:serviceaccount:default:default\\\" cannot get resource \\\"endpoints\\\" in API group \\\"\\\" in the namespace \\\"default\\\"\",\"reason\":\"Forbidden\",\"details\":{\"name\":\"emqx-service\",\"kind\":\"endpoints\"},\"code\":403}\n"}
···

Pod is refused because of authority, when it accesses Kubernetes apiserver and return HTTP 403, the cluster failed.

The normal pod can not access Kubernetes apiserver. There are two methods that can figure out this problem. The first one is to open the HTTP interface of Kubernetes apiserver, but some security risks are exiting. Another one is through ServiceAccount, Role and RoleBinding to configure RBAC authentication.

  • Define ServiceAccount, Role and RoleBinding:

    $ cat rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: emqx
---
kind: Role
apiVersion: rbac.authorization.kubernetes.io/v1beta1
metadata:
  namespace: default
  name: emqx
rules:
- apiGroups:
  - ""
  resources:
  - endpoints 
  verbs: 
  - get
  - watch
  - list
---
kind: RoleBinding
apiVersion: rbac.authorization.kubernetes.io/v1beta1
metadata:
  namespace: default
  name: emqx
subjects:
  - kind: ServiceAccount
    name: emqx
    namespace: default
roleRef:
  kind: Role
  name: emqx
  apiGroup: rbac.authorization.kubernetes.io

  • Deploy the corresponding resources:

    $ kubectl apply -f rbac.yaml
serviceaccount/emqx created
role.rbac.authorization.kubernetes.io/emqx created
rolebinding.rbac.authorization.kubernetes.io/emqx created

  • Modify file yaml of Deployment, add spec.template.spec.serviceAccountName, and redeploy:

    $cat deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: emqx-deployment
  labels:
    app: emqx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      serviceAccountName: emqx
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083
        env:
        - name: EMQX_NAME
          value: emqx
        - name: EMQX_CLUSTER__DISCOVERY
          value: kubernetes
        - name: EMQX_CLUSTER__K8S__APP_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__SERVICE_NAME
          value: emqx-service
        - name: EMQX_CLUSTER__K8S__APISERVER
          value: "https://kubernetes.default.svc:443"
        - name: EMQX_CLUSTER__K8S__NAMESPACE
          value: default

$ kubectl delete deployment emqx-deployment
deployment.apps "emqx-deployment" deleted

$ kubectl apply -f deployment.yaml
deployment.apps/emqx-deployment created

  • View status:

    $ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
emqx-deployment-6b854486c-dhd7p   1/1     Running   0          10s
emqx-deployment-6b854486c-psv2r   1/1     Running   0          10s
emqx-deployment-6b854486c-tdzld   1/1     Running   0          10s

$ kubectl exec emqx-deployment-6b854486c-dhd7p  -- emqx_ctl status
Node 'emqx@192.168.77.92' is started
emqx 4.1-rc.1 is running

$ kubectl exec emqx-deployment-6b854486c-dhd7p  -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@192.168.77.115','emqx@192.168.77.92',
                       'emqx@192.168.87.157'],
                  stopped_nodes => []}

  • abort a Pod:

    $ kubectl delete pods emqx-deployment-6b854486c-dhd7p
pod "emqx-deployment-6b854486c-dhd7p" deleted

$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
emqx-deployment-6b854486c-846v7   1/1     Running   0          56s
emqx-deployment-6b854486c-psv2r   1/1     Running   0          3m50s
emqx-deployment-6b854486c-tdzld   1/1     Running   0          3m50s

$ kubectl exec emqx-deployment-6b854486c-846v7 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@192.168.77.115','emqx@192.168.77.84',
                       'emqx@192.168.87.157'],
                  stopped_nodes => ['emqx@192.168.77.92']}

    The output result represents that EMQX Broker will correctly display the stopped pod, and add the new created Pod to the cluster.

So far, EMQX Broker has successfully created cluster in kubernetes.

Persistence EMQX Broker cluster

The Deployment used above to manage Pod, but the network of Pod is constantly changeable. When the pod is destroyed and rebuild, the data and configuration stored in EMQX Broker also disappeared, this is can not be accepted during production. Next try to persistence EMQX Broker cluster. Even if the pod is destroyed and rebuild, the data of the EMQX Broker can be retained.

ConfigMap

ConfigMap is an API object which is used to store non-confidential data to key-value pairs. It will be used as an environment variables, command-line arguments, or as configuration files in a volume.

ConfigMap decouples your environment configuration information from container mirroring, so that you can easily modify application configuration.

ConfigMap does not provide secrecy or encryption. If the data you want to store are confidential, use a Secret rather than a ConfigMap, or use third-party tools to keep your data private.

Next use ConfigMap to record the configuration of EMQX Broker, and import them as environment variables to the Deployment.

  • Define Configmap and deploy:

    $cat configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: emqx-config
data:
  EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname"
  EMQX_CLUSTER__K8S__APISERVER: "https://kubernetes.default.svc:443"
  EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local"

$ kubectl apply -f configmap.yaml
configmap/emqx-config created

  • Configure Deployment for using Configmap

    $cat deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: emqx-deployment
  labels:
    app: emqx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      serviceAccountName: emqx
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083
        envFrom:
          - configMapRef:
              name: emqx-config

  • Redeploy Deployment and view status

    $ kubectl delete -f deployment.yaml
deployment.apps "emqx-deployment" deleted

$ kubectl apply -f deployment.yaml
deployment.apps/emqx-deployment created

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
emqx-deployment-5c7696b5d7-k9lzj   1/1     Running   0          3s
emqx-deployment-5c7696b5d7-mdwkt   1/1     Running   0          3s
emqx-deployment-5c7696b5d7-z57z7   1/1     Running   0          3s

$ kubectl exec emqx-deployment-5c7696b5d7-k9lzj -- emqx_ctl status
Node 'emqx@192.168.87.149' is started
emqx 4.1-rc.1 is running

$ kubectl exec emqx-deployment-5c7696b5d7-k9lzj -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@192.168.77.106','emqx@192.168.77.107',
                       'emqx@192.168.87.149'],
                  stopped_nodes => []}

The configuration files of EMQX Broker have decoupled to Configmap. If necessary, you can freely configure one or more Configmap, and import them as environment variables or files to the pod.

StatefulSet

StatefulSet is used for figuring out the problem that is stateful service(Deployments and ReplicaSets are designed for stateless service). The scenarios it is applied including:

  • Stable persistence storage, that is, after the pod is re-dispatched, it can also access the same persistence data. Based on PVC to implement.
  • Stable network sign, that is, after the pod is re-dispatched, it's PodName and HostName have no changes. Based on Headless Service(the Service without Cluster IP) to implement.
  • Orderly deployment, orderly expands, that is, the pod is sequential. So that it must be carried out in sequence according to the defined order when deploying or expanding(from 0 to N-1, all the previous pods must are Running and Ready before running the next pod). Based on init containers to implement.
  • Orderly shrink, orderly delete(from 0 to N-1)

From the above scenarios, we can find that StatefulSet consists of the following parts:

  • Headless Service for defining network sign(DNS domain)
  • VolumeClaimTemplates for creating PersistentVolumes
  • StatefulSet for defining specific applications

The DNS format of every pod in StatefulSet is statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local:

  • serviceName is the name of Headless Service
  • 0..N-1 is the number of Pod, from 0 to N-1
  • statefulSetName is the name of StatefulSet
  • namespace is the namespace where service locate in, Headless Service and StatefulSet must locate in the same namespace
  • .cluster.local is Cluster Domain

Then, using StatefulSet replace Deployment to manage pod.

  • Delete Deployment:

    $ kubectl delete deployment emqx-deployment
deployment.apps "emqx-deployment" deleted

  • Define StatefulSet:

    $cat statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: emqx-statefulset
  labels:
    app: emqx
spec:
  serviceName: emqx-headless
  updateStrategy:
    type: RollingUpdate
  replicas: 3
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      serviceAccountName: emqx
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083
        envFrom:
          - configMapRef:
              name: emqx-config

    It should be noted that StatefulSet needs Headless Service to implement a stable network sign, so we need to define one Service more.

    $cat headless.yaml

apiVersion: v1
kind: Service
metadata:
  name: emqx-headless
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: emqx
  ports:
  - name: mqtt
    port: 1883
    protocol: TCP
    targetPort: 1883
  - name: mqttssl
    port: 8883
    protocol: TCP
    targetPort: 8883
  - name: mgmt
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: websocket
    port: 8083
    protocol: TCP
    targetPort: 8083
  - name: wss
    port: 8084
    protocol: TCP
    targetPort: 8084
  - name: dashboard
    port: 18083
    protocol: TCP
    targetPort: 18083

    Because Headless Service does not require an IP, configured clusterIP: None.

  • Deploy the corresponding configurations:

    $ kubectl apply -f headless-service.yaml
service/emqx-headless created

$ kubectl apply -f statefulset.yaml
statefulset.apps/emqx-deployment created

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
emqx-statefulset-0                 1/1     Running   0          2m59s
emqx-statefulset-1                 1/1     Running   0          2m57s
emqx-statefulset-2                 1/1     Running   0          2m54s

$ kubectl exec emqx-statefulset-0 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@192.168.77.105','emqx@192.168.87.153',
                       'emqx@192.168.87.155'],
                  stopped_nodes => []}

  • Update Configmap:

    StatefulSet provides a stable network sign. EMQX Broker supports using hostname and DNS rules to replace IP for implementing cluster. Taking hostname as an example, need to modify emqx.conf:

    cluster.kubernetes.address_type = hostname
cluster.kubernetes.suffix = "svc.cluster.local"

    The DNS rules of Pod in the cluster Kubernetes can be customized by users. EMQX Broker provides cluster.kubernetes.suffix for facilitating users to match custom DNS rules. This article use the default DNS rules: statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local. serviceName in the DNS rules is Headless Service used by StatefulSet, so we also need to modify cluster.kubernetes.service_name to Headless Service Name.

    Convert configuration items to environment variables, we need to configure the following items in the Configmap:

    EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname"
EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local"
EMQX_CLUSTER__K8S__SERVICE_NAME: emqx-headless

    Configmap provides hot update. Execute $ kubectl edit configmap emqx-config to a hot update Configmap.

  • Redeploy StatefulSet:

    Pod will not be re-enabled after updating Configmap, so we need to manually update StatefulSet

    $ kubectl delete statefulset emqx-statefulset
statefulset.apps "emqx-statefulset" deleted

$ kubectl apply -f statefulset.yaml
statefulset.apps/emqx-statefulset created

$ kubectl get pods
NAME                 READY   STATUS    RESTARTS   AGE
emqx-statefulset-0   1/1     Running   0          115s
emqx-statefulset-1   1/1     Running   0          112s
emqx-statefulset-2   1/1     Running   0          110s

$ kubectl exec emqx-statefulset-2 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local',
                       'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local',
                       'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'],
                  stopped_nodes => []}

    We can see that the new EMQX Broker cluster has successfully built.

  • Abort a Pod:

    The PodName and HostName of Pod in the StatefulSet will not change after rescheduling. Let's try:

    $ kubectl get pods
kuNAME                 READY   STATUS    RESTARTS   AGE
emqx-statefulset-0   1/1     Running   0          6m20s
emqx-statefulset-1   1/1     Running   0          6m17s
emqx-statefulset-2   1/1     Running   0          6m15s

$ kubectl delete pod emqx-statefulset-0
pod "emqx-statefulset-0" deleted

$ kubectl get pods
NAME                 READY   STATUS    RESTARTS   AGE
emqx-statefulset-0   1/1     Running   0          27s
emqx-statefulset-1   1/1     Running   0          9m45s
emqx-statefulset-2   1/1     Running   0          9m43s

$ kubectl exec emqx-statefulset-2 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local',
                       'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local',
                       'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'],
                  stopped_nodes => []}

    As expected, StatefulSet rescheduled a pod with the same network symbol. EMQX Broker in the pod has also successfully joined the cluster.

StorageClasses, PersistentVolume and PersistentVolumeClaim

PersistentVolume(PV) is the storage set by the administrator, and it is part of the cluster. It like the node is the resource of the cluster, PV is also the resource od cluster. PV is volume plugin like Volume, but it has a life cycle which is independent of the pod using PV. This API object includes the details of the implementation of storage, that is, NFS, iSCSI or storage systems specific to cloud vendors.

PersistentVolumeClaim(PVC) is the requirements stored by users. It is similar to Pod. Pod consumes the node resource, and PVC consumes PV resources. Pod can request resources at a specific level(CPU and RAM). The declaration can request the specific size and accessing mode(for example, can read/write once or mount in read-only multiple)

StorageClass describes the method used to store "class" for administrators. Different class may be mapped to different quality of service levels or backup strategies, or the arbitrary strategy confirmed by the cluster administrator. Kubernetes itself is not clear what the various class represent. This concept is sometimes referred to as "configuration file" in other storage systems.

You can create PV or StorageClass in advance when deploy EMQX Broker, and then using PVC mount the directory /opt/emqx/data/mnesia of EMQX Broker. After rescheduling Pods, EMQX will recover the data from directory /opt/emqx/data/mnesia for implementing the persistence of EMQX Broker.

  • Define StatefulSet

    $cat statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: emqx-statefulset
  labels:
    app: emqx
spec:
  replicas: 3
  serviceName: emqx-headless
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      volumes:
      - name: emqx-data
        persistentVolumeClaim:
          claimName: emqx-pvc
      serviceAccountName: emqx
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083
        envFrom:
          - configMapRef:
              name: emqx-config
        volumeMounts:
        - name: emqx-data
          mountPath: "/opt/emqx/data/mnesia"
  volumeClaimTemplates:
  - metadata:
      name: emqx-pvc
      annotations:
        volume.alpha.kubernetes.io/storage-class: manual
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

    First, this file specifies using the storage class named manual through volumeClaimTemplates to create the PVC resources named emqx-pvc. The read and write mode of PVC resources is ReadWriteOnce, which need 1Gi space. Next, define this PVC as a volume named emqx-data, and mount this volume on the directory /opt/emqx/data/mnesia of Pod.

  • Deploy resources:

    Users or cluster Kubernetes administrators need to create storage class before deploying StatefulSet.

    $ kubectl apply -f statefulset.yaml
statefulset.apps/emqx-statefulset created

$ kubectl get pods
NAME                 READY   STATUS    RESTARTS   AGE
emqx-statefulset-0   1/1     Running   0          27s
emqx-statefulset-1   1/1     Running   0          9m45s
emqx-statefulset-2   1/1     Running   0          9m43s

$ kubectl get pvc
NAME                                 STATUS    VOLUME                                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
emqx-data-emqx-statefulset-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            2m11s
emqx-data-emqx-statefulset-0   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            99s
emqx-data-emqx-statefulset-0   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            56s

    The output result represents that the status of this PVC is Bound, and PVC storage has been successfully established. EMQX Broker will read the data is mounted in PVC for implementing persistence, when rescheduling pod.

So far, the process that EMQX Broker builds persistence cluster in the Kubernetes has been completed. This article omits some details, and the process of deployment is also for the simple Demo. Users can read kubernetes documentation and Helm chart source code provided by the EMQX Team for more in-depth research. Of course, contributing issues, pulling requests and star on Github are welcome.

Try EMQX Cloud for Free
No credit card required
Get Started →
Installation
Edit
EMQX Team

EMQX Team

The EMQX Team specializes in developing MQTT messaging infrastructure and solutions, including the most scalable open-source MQTT broker on the market – EMQX. The team also coordinates excellence on the enterprise MQTT platform at scale – EMQX Enterprise.

Subscribe to our blogs

Related Posts

Jan 27, 2021Zhiwei Yu
Build an EMQX cluster based on HAProxy

HAProxy can provide high availability, load balancing, and TCP and HTTP based application proxies. This article will introduce how to build the EMQX cluster based on HAProxy.

Mar 15, 2023EMQX Team
How to Gracefully Upgrade EMQX on Kubernetes

Combines Node Evacuation with simulated blue-green deployment, effectively addressing common issues such as multiple disconnections, potential service overload, and load imbalance during upgrades.