Topology Operator

The Topology Operator method of MATRIXX Engine pod management utilizes multiple custom resources (CRs) and operator pods working together to manage the installation and upgrade of multiple sub-domains and engines across Kubernetes clusters, including the loading of pricing and taxation data.

Important: The Engine Operator method of engine management is deprecated in this release and will be removed in a future release of MATRIXX.
Topology Operator Components describes the components that make up Topology Operator.
Table 1. Topology Operator Components
Pod or Custom Resource Description
MtxTopology CR A CR that has information for all engines and pricing. This is created and updated by Helm and watched by Topology Operator.
MtxSubdomain CR A CR that has the information for all engines and pricing in a sub-domain. MtxSubdomain CRs are created, updated, or deleted by Topology Operator and watched by a subdomain-operator.
MtxEngine CR A CR that has the information for a single engine. MtxEngine CRs are created, updated, or deleted by the subdomain-operator corresponding to that sub-domain and watched by an Engine Operator.
topology-operator Topology Operator watches the MtxTopology CR. It identifies the corresponding MtxSubdomain CRs and compares them with those that exist. The topology-operator pod performs all necessary create, update, or delete MtxSubdomain CR actions.
subdomain-resource-operator The subdomain-resource-operator creates, updates, or deletes subdomain-level resources for each MtxSubdomain CR, such as subdomain-operator and pricing-operator pods.
engine-resource-operator The engine-resource-operator creates, updates, or deletes engine-level resources for each MtxEngine CR, such as engine-operator pods (using topology-agent instances).
subdomain-operator One subdomain-operator pod per sub-domain watches the MtxSubdomain CR for that sub-domain, identifies corresponding MtxEngine CRs, and compares them with those that exist.

Subdomain-operators identify any needed MtxEngine CR creations, updates, or deletions and perform those operations (using topology-agent instances) in order, waiting for one step to finish before starting the next.
pricing-operator One pricing-operator instance per sub-domain watches the MtxSubdomain CR to determine if an engine is ready for pricing to be loaded and creates pricing-loader instances (using topology-agent instances) as necessary.
tax-operator One tax-operator instance per sub-domain watches the MtxSubdomain CR to determine if an engine is ready for tax to be loaded and creates tax-loader instances (using topology-agent instances) as necessary.
subdomain-health-checker A subdomain-health-checker instance per sub-domain includes the following:
  • Engine communication health check.
  • GTC sync health check.
  • Sub-domain health check brain.
topology-agent A topology-agent in each cluster performs actions as delegated by topology-operator, subdomain-operator, and pricing-operator.
engine-operator One engine-operator instance per engine watches the MtxEngine CR for that engine. The engine-operator creates an engine-stopper instance if required during an update. It creates or updates child resources, creates the pod-monitor, and creates the engine-starter instance when pod-monitor reports all pods are running.
engine-health-checker An engine-health-checker instance per engine contains the following:
  • Cluster Monitor
  • Engine System Health Checker
pod-monitor A pod-monitor for each engine watches all child pods of that engine and communicates with the engine-operator to let it know when all pods are running and the engine can be started.
engine-starter, engine-stopper Instances that start and stop engines, replacing the same functionality in the mgr-engine images.
pricing-loader A pricing-loader instance loads pricing on an engine.
tax-loader A tax-loader instance loads tax on an engine.

Installation

During installation of a Topology Operator-based MATRIXX deployment, Helm creates one MtxTopology CR and one topology-operator instance. The Topology Operator compares the current MtxSubdomain CRs in the cluster (of which there are initially none) with the target MtxSubdomain CRs based on the MtxTopology CR. The Topology Operator then creates all MtxSubdomain CRs.

The subdomain-resource-operator creates various sub-domain resources including one subdomain-operator-sX, one pricing-operator-sX, one tax-operator-sX, and one subdomain-health-checker-sX per MtxSubdomain CR. The subdomain-operator-sX instance compares the current MtxEngine CRs related to its sub-domain in the cluster(s) (of which initially there are none) with the target MtxEngine CRs based on the MtxSubdomain CR. The subdomain-operator-sX then creates MtxEngine CRs one at a time, waiting until the engine is handled by its engine-operator-sXeY before going to the next.

The engine-resource-operator creates various engine-related resources including one engine-operator-sXeY and one engine-health-checker-sXeY per MtxEngine CR. The engine-operator-sXeY creates child resources and pod-monitor-sXeY instances for auto-healing. When pod-monitor-sXeY reports that the engine is ready to be started, engine-operator creates an engine-starter-sXeY.

A pricing-operator-sX handles the loading of pricing on the active engine in a sub-domain by creating instances of pricing-loader, and a tax-operator-sX handles the loading of tax on the active engine in a sub-domain by creating instances of tax-loader-sXeY.

The topology-operator, subdomain-operator-sX, pricing-operator-sX, tax-operator-sX, and subdomain-health-checker-sX instances are collectively referred to as master operators. In each cluster, topology-agent instances act on behalf of the masters to create, update, and delete resources on different clusters where those components lack permissions to perform these operations.

Upgrade

A Topology Operator-based deployment is upgraded using the helm upgrade command. The helm upgrade command creates a pre-update instance that scales all topology-operator, subdomain-operator, pricing-operator, tax-operator, and engine-operator deployments to zero replicas. Once this has completed, all deployments are scaled back up to one replica. This is to ensure that only the updated operators handle the upgrade.

Helm updates the MtxTopology CR. The Topology Operator compares the current MtxSubdomain CRs in the cluster with the target MtxSubdomain CRs based on the MtxTopology CR. The topology-operator then updates all MtxSubdomain CRs as necessary.

The subdomain-operator-sX compares the current MtxEngine CRs related to its sub-domain in the cluster(s) with the target MtxEngine CRs based on the MtxSubdomain CR. The subdomain-operator-sX then updates MtxEngine CRs as necessary, one at a time, starting with the standby engine, waiting until the engine is handled by its engine-operator-sXeY instance before going to the next.

The engine-operator-sXeY instance observes that the MtxEngine CR has been updated and handles this by stopping the engine with an engine-stopper-sXeY instance. It then updates engine child resources and waits for the pod-monitor-sXeY to determine that the engine is ready to be restarted before creating an engine-starter-sXeY.

Similarly, the pricing-operator-sX instance handles the loading of new pricing data on the active engine in the sub-domain by creating pricing-loader-sXeY instances. A tax-operator-sX instance handles the loading of new taxation data on the active engine in the sub-domain by creating instances of tax-loader-sXeY.

This same procedure is used for configuration, maintenance, and software upgrades. It can also be used to add or remove sub-domains or engines. It also facilitates rolling back using the helm rollback command.

Uninstall

Uninstalling with Helm involves a cascading sequence similar to the one used during installation. The helm uninstall command creates a pre-delete instance that marks the MtxTopology CR for deletion. When the Topology Operator sees that the MtxTopology CR is marked for deletion, it marks all MtxSubdomain CRs for deletion. A subdomain-operator-sX sees that its MtxSubdomain CR is marked for deletion and marks all MtxEngine CRs related to its sub-domain for deletion.

When each engine-operator-sXeY sees that its MtxEngine CR is marked for deletion, it first stops the engine using an engine-stopper-sXeY and then deletes the engine resources it created. When this process is finished, the engine-resource-operator deletes engine-related resources that it created. When this process is finished, the MtxEngine CR is finally deleted, triggering the deletion of the remaining engine-related resources.

Once all MtxEngine CRs related to a sub-domain are deleted, the subdomain-resource-operator deletes sub-domain-related resources that it created. When this process is finished the MtxSubdomain CR is finally deleted, triggering the deletion of the remaining sub-domain-related resources.

After all MtxSubdomain CRs are deleted, the MtxTopology CR is finally deleted. The pre-delete completes and the helm uninstall command finally deletes the topology-operator pod itself and other Helm-created resources.