Install in Multiple Clusters with Dynamic Provisioning

Deploy across multiple clusters with dynamic provisioning by creating the load balancer services with the required annotations, allowing the IP addresses to be dynamically provisioned. Then update your configuration with the assigned IP addresses.

Procedure

  1. Create the namespaces with the following commands: 
    kubectl --context context1 create ns matrixx-operators
kubectl --context context2 create ns matrixx-operators
kubectl --context context1 create ns matrixx-engine-s1
kubectl --context context2 create ns matrixx-engine-s1
  2. Install in cluster 1 namespace matrixx-engine-s1 with the following command: 
    helm --kube-context context1 install mtx-engine-s1 matrixx/matrixx -n matrixx-engine-s1 -f base.yaml -f topology.yaml -f cluster1.yaml --version matrixx_version

    Where the topology.yaml Helm values file has the following contents:

    global:
  topology:
    operators:
      master:
        context: context1
        namespace: matrixx-operators
      agents:
      - context: context1
        namespace: matrixx-operators
        externalAddress: dynamic
        serviceAnnotations:
          topology-agent-1_service_annotations
        useDynamicIPs: true
      - context: context2
        namespace: matrixx-operators
        externalAddress: dynamic
        serviceAnnotations:
          topology-agent-2 service annotations
        useDynamicIPs: true
    domains:
    - subdomains:
      - engines:
        - context: context1
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: dynamic
              udp: dynamic
            serviceAnnotations:
              tcp:
                proc-cls-s1e1-tcp service annotations
              udp:
                proc-cls-s1e1-udp service annotations
          publishing:
            externalAddress:
              tcp: dynamic
              udp: dynamic
            serviceAnnotations:
              tcp:
                publ-cls-s1e1-tcp service annotations
              udp:
                publ-cls-s1e1-udp service annotations
          useDynamicIPs: true
        - context: context2
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: dynamic
              udp: dynamic
            serviceAnnotations:
              tcp:
                proc-cls-s1e2-tcp service annotations
              udp:
                proc-cls-s1e2-udp service annotations
          publishing:
            externalAddress:
              tcp: dynamic
              udp: dynamic
            serviceAnnotations:
              tcp:
                publ-cls-s1e2-tcp service annotations
              udp:
                publ-cls-s1e2-udp service annotations
          useDynamicIPs: true
    Note: Initially we configure all external addresses as dynamic and set useDynamicIPs to true. The actual external address values are not important at this stage. Use of the dynamic string is arbitrary. The only requirement is that some value must be set to pass Helm validation and allow the install command to succeed. IP addresses dynamically provisioned at a later stage will be used to update this file.
    Helm creates load balancer services in cluster 1 namespace matrixx-s1e1 with dynamically provisioned IP addresses. As load balancers for engine s1e2 are not deployed yet, the external name services for s1e2 have a value of dynamic for externalName. You update these after deploying engine s1e2.
  3. Install in cluster 2 namespace mtx-engine-s1 with the following command: 
    helm --kube-context context2 install mtx-engine-s1 matrixx/matrixx -n matrixx-engine-s1 -f base.yaml -f topology.yaml -f cluster2.yaml --version matrixx_version

    Where topology.yaml is updated to have the following contents:

    global:
  topology:
    operators:
      master:
        context: context1
        namespace: matrixx-operators
      agents:
      - context: context1
        namespace: matrixx-operators
        externalAddress: dynamic
        serviceAnnotations:
          topology-agent-1 service annotations
        useDynamicIPs: true
      - context: context2
        namespace: matrixx-operators
        externalAddress: dynamic
        serviceAnnotations:
          topology-agent-2 service annotations
        useDynamicIPs: true
    domains:
    - subdomains:
      - engines:
        - context: context1
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: proc-cls-s1e1-tcp external address
              udp: proc-cls-s1e1-udp external address
            serviceAnnotations:
              tcp:
                proc-cls-s1e1-tcp service annotations
              udp:
                proc-cls-s1e1-udp service annotations
          publishing:
            externalAddress:
              tcp: publ-cls-s1e1-tcp external address
              udp: publ-cls-s1e1-udp external address
            serviceAnnotations:
              tcp:
                publ-cls-s1e1-tcp service annotations
              udp:
                publ-cls-s1e1-udp service annotations
        - context: context2
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: dynamic
              udp: dynamic
            serviceAnnotations:
              tcp:
                proc-cls-s1e2-tcp service annotations
              udp:
                proc-cls-s1e2-udp service annotations
          publishing:
            externalAddress:
              tcp: dynamic
              udp: dynamic
            serviceAnnotations:
              tcp:
                publ-cls-s1e2-tcp service annotations
              udp:
                publ-cls-s1e2-udp service annotations
    Helm creates load balancer services in cluster 2 namespace matrixx-engine-s1 with dynamically provisioned IP addresses. Update topology.yaml with these IP addresses. Helm also creates external name services for s1e1 in cluster 2 namespace matrixx-engine-s1 which correctly point to the load balancer services for s1e1 in cluster 1 namespace matrixx-engine-s1.
  4. Upgrade in cluster 1 namespace matrixx-engine-s1 with the following command: 
    helm --kube-context context1 upgrade mtx-engine-s1 matrixx/matrixx -n matrixx-engine-s1 -f base.yaml -f topology.yaml -f cluster1.yaml --version matrixx_version

    Where an updated topology.yaml has the following contents:

    global:
  topology:
    operators:
      master:
        context: context1
        namespace: matrixx-operators
      agents:
      - context: context1
        namespace: matrixx-operators
        externalAddress: dynamic
        serviceAnnotations:
          topology-agent-1 service annotations
        useDynamicIPs: true
      - context: context2
        namespace: matrixx-operators
        externalAddress: dynamic
        serviceAnnotations:
          topology-agent-2 service annotations
        useDynamicIPs: true
    domains:
    - subdomains:
      - engines:
        - context: context1
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: proc-cls-s1e1-tcp external address
              udp: proc-cls-s1e1-udp external address
            serviceAnnotations:
              tcp:
                proc-cls-s1e1-tcp service annotations
              udp:
                proc-cls-s1e1-udp service annotations
          publishing:
            externalAddress:
              tcp: publ-cls-s1e1-tcp external address
              udp: publ-cls-s1e1-udp external address
            serviceAnnotations:
              tcp:
                publ-cls-s1e1-tcp service annotations
              udp:
                publ-cls-s1e1-udp service annotations
        - context: context2
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: proc-cls-s1e2-tcp external address
              udp: proc-cls-s1e2-udp external address
            serviceAnnotations:
              tcp:
                proc-cls-s1e2-tcp service annotations
              udp:
                proc-cls-s1e2-udp service annotations
          publishing:
            externalAddress:
              tcp: proc-cls-s1e2-tcp external address
              udp: proc-cls-s1e2-udp external address
            serviceAnnotations:
              tcp:
                publ-cls-s1e2-tcp service annotations
              udp:
                publ-cls-s1e2-udp service annotations
    Helm updates the external name services for s1e2 in cluster 1 namespace matrixx-engine-s1 to correctly point to the load balancer services for s1e2 in cluster 2 namespace matrixx-engine-s1.
  5. Install in cluster 2 matrixx-operators namespace using the topology.yaml file from step 4 with the following command: 
    helm --kube-context context2 install mtx-operators matrixx/matrixx -n matrixx-operators -f base.yaml -f topology.yaml -f cluster2.yaml --version matrixx_version
    Helm creates topology-agent load balancer service in cluster 2 namespace matrixx-operators and an IP address is provisioned for the topology-agent-2 external address service. Use this value to update topology.yaml.
  6. Install in the cluster 1 matrixx-operators namespace with the following command: 
    helm --kube-context context1 install mtx-operators matrixx/matrixx -n matrixx-operators -f base.yaml -f topology.yaml -f cluster1.yaml --version matrixx_version

    Where topology.yaml has the following contents:

    global:
  topology:
    operators:
      master:
        context: context1
        namespace: matrixx-operators
      agents:
      - context: context1
        namespace: matrixx-operators
        externalAddress: dynamic
        serviceAnnotations:
          topology-agent-1 service annotations
        useDynamicIPs: true
      - context: context2
        namespace: matrixx-operators
        externalAddress: topology-agent-2 external address
        serviceAnnotations:
          topology-agent-2 service annotations
    domains:
    - subdomains:
      - engines:
        - context: context1
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: proc-cls-s1e1-tcp external address
              udp: proc-cls-s1e1-udp external address
            serviceAnnotations:
              tcp:
                proc-cls-s1e1-tcp service annotations
              udp:
                proc-cls-s1e1-udp service annotations
          publishing:
            externalAddress:
              tcp: publ-cls-s1e1-tcp external address
              udp: publ-cls-s1e1-udp external address
            serviceAnnotations:
              tcp:
                publ-cls-s1e1-tcp service annotations
              udp:
                publ-cls-s1e1-udp service annotations
        - context: context2
          namespace:  matrixx-engine-s1
          processing:
            externalAddress:
              tcp: proc-cls-s1e2-tcp external address
              udp: proc-cls-s1e2-udp external address
            serviceAnnotations:
              tcp:
                proc-cls-s1e2-tcp service annotations
              udp:
                proc-cls-s1e2-udp service annotations
          publishing:
            externalAddress:
              tcp: publ-cls-s1e2-tcp external address
              udp: publ-cls-s1e2-udp external address
            serviceAnnotations:
              tcp:
                publ-cls-s1e2-tcp service annotations
              udp:
                publ-cls-s1e2-udp service annotations
    Helm creates external name services for the topology-agents in cluster 1 namespace matrixx-operators and an IP address is provisioned for the topology-agent-1 external address service. Use this value to update topology.yaml.
  7. Upgrade in cluster 1 namespace matrixx-operators with the following command: 
    helm --kube-context context1 upgrade mtx-operators matrixx/matrixx -n matrixx-operators -f base.yaml -f topology.yaml -f cluster1.yaml --version matrixx_version

    Where topology.yaml has the following contents:

    global:
  topology:
    operators:
      master:
        context: context1
        namespace: matrixx-operators
      agents:
      - context: context1
        namespace: matrixx-operators
        externalAddress: topology-agent-1 external address
        serviceAnnotations:
          topology-agent-1 service annotations
      - context: context2
        namespace: matrixx-operators
        externalAddress: topology-agent-2 external address
        serviceAnnotations:
          topology-agent-2 service annotations
    domains:
    - subdomains:
      - engines:
        - context: context1
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: proc-cls-s1e1-tcp external address
              udp: proc-cls-s1e1-udp external address
            serviceAnnotations:
              tcp:
                proc-cls-s1e1-tcp service annotations
              udp:
                proc-cls-s1e1-udp service annotations
          publishing:
            externalAddress:
              tcp: publ-cls-s1e1-tcp external address
              udp: publ-cls-s1e1-udp external address
            serviceAnnotations:
              tcp:
                publ-cls-s1e1-tcp service annotations
              udp:
                publ-cls-s1e1-udp service annotations
        - context: context2
          namespace: matrixx-engine-s1
          processing:
            externalAddress:
              tcp: proc-cls-s1e2-tcp external address
              udp: proc-cls-s1e2-udp external address
            serviceAnnotations:
              tcp:
                proc-cls-s1e2-tcp service annotations
              udp:
                proc-cls-s1e2-udp service annotations
          publishing:
            externalAddress:
              tcp: publ-cls-s1e2-tcp external address
              udp: publ-cls-s1e2-udp external address
            serviceAnnotations:
              tcp:
                publ-cls-s1e2-tcp service annotations
              udp:
                publ-cls-s1e2-udp service annotations