Home
MATRIXX Installation and Upgrade
Topology Operator
The Topology Operator method of MATRIXX Engine pod management utilizes multiple custom resources (CRs) and operator pods working together to manage the installation and upgrade of multiple sub-domains and engines across Kubernetes clusters, including the loading of pricing and taxation data.
Adding or Removing an Engine or Sub-Domain
Adding a MATRIXX Engine or sub-domain is similar to the processes used for configuration, maintenance, or software upgrades. This should be performed in the following order:
Removing a Sub-Domain
When removing a sub-domain, upgrading the master namespace must be done in two steps.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
- About MATRIXX Installation and Upgrade
  MATRIXX Installation and Upgrade describes the tasks required to install MATRIXX components as containers using Kubernetes and Helm.
- Cloud Native Infrastructure Requirements
  Cloud native MATRIXX deployments have infrastructure requirements for third-party software versions, Kubernetes pod characteristics, and container characteristics. Nodes running MATRIXX components have operating system, memory, networking, and storage requirements.
- Topology Operator and Engine Operator Feature Differences
  Different MATRIXX features are supported depending on whether your installation is based on Topology Operator or Engine Operator.
- Installing MATRIXX
  A cloud native MATRIXX installation consists of containers installed with the MATRIXX Helm chart in a multi-node Kubernetes cluster (or clusters), requiring MATRIXX images and MATRIXX Engine custom resource definitions and controllers.
- Topology Operator Installation and Upgrade Examples
  These examples show how you can configure the installation of multiple MATRIXX Engines in a single namespace, multiple namespaces in a single cluster, and multiple clusters.
- Configuring MATRIXX
  MATRIXX is configured using Helm properties. Questions and answers in create_config.info files can also provide configuration using configuration sources. For information about how to configure MATRIXX, see the discussions of those topics in MATRIXX Configuration.
- Topology Operator
  The Topology Operator method of MATRIXX Engine pod management utilizes multiple custom resources (CRs) and operator pods working together to manage the installation and upgrade of multiple sub-domains and engines across Kubernetes clusters, including the loading of pricing and taxation data.
  - Adding or Removing an Engine or Sub-Domain
    Adding a MATRIXX Engine or sub-domain is similar to the processes used for configuration, maintenance, or software upgrades. This should be performed in the following order:
    - Initial Deployment
      A hypothetical, multi-namespace, single sub-domain deployment has the following distribution of components across namespaces:
    - Adding an Engine
      Add a third MATRIXX Engine (in the same sub-domain) to the initial deployment in its own namespace:
    - Removing an Engine
      When removing a MATRIXX Engine (s1e3) from a three-engine deployment (s1e1, s1e2, and s1e3), upgrading the master namespace is a two-step process.
    - Adding a Sub-Domain
      Adding a second sub-domain (with two MATRIXX Engines) to the initial deployment results in the following distribution of components across namespaces:
    - Removing a Sub-Domain
      When removing a sub-domain, upgrading the master namespace must be done in two steps.
  - Health Checks
    The subdomain-health-checker-sX and engine-health-checker-sXeY pods run multiple containers, each of which is a separate health check.
  - Disaster Recovery
    In multi-cluster installations, if a cluster fails, Topology Operator recovers in different ways depending on whether the cluster contained master instances or not.
  - Changes to Custom Resource Definitions
    MtxSubdomain and MtxEngine custom resource definitions (CRDs) for Pricing Controller, Engine Controller, and Engine Operator-based deployments are in the matrixx.com group. MtxTopology, MtxSubdomain, and MtxEngine CRDs for use with Topology Operator-based deployments are in the matrixx.matrixx.com group.
  - Manually Starting and Stopping Engines
    Do not use the stop_engine and start_engine scripts with Topology Operator-based deployments. Instead, to start or stop MATRIXX Engine, patch the MtxEngine CR. Start and stop actions are then handled by the operators.
  - Scaling MATRIXX Engine Pods with Helm
    In Topology Operator-based installations, increase or decrease the MATRIXX Engine pod replica count with the helm upgrade command, in the same way as any other configuration change.
  - Reconciliation Errors
    Sometimes errors may be seen in the logs of the various different operators as they try to reconcile the current state and target state. In certain cases these errors can be ignored.
  - Uninstalling a Topology Operator-Based Deployment
    To uninstall a Topology Operator-based deployment, uninstall the release, then delete the namespace.
  - Topology Operator Configuration
    Configure MATRIXX deployments based on Topology Operator by adding values for properties from the MATRIXX Helm chart to your Helm values files.
  - Topology Operator Pod Annotations
    Topology Operator components have pod annotations available to match, including annotations added to preexisting MATRIXX.
- Scaling MATRIXX
  Use Helm and Kubernetes to manually add or remove pods as necessary. You can also use Kubernetes to manage resources available to MATRIXX Engine pods.
- Network Enablers in a Kubernetes Cluster
  For Call Control Framework (CCF), only the deployment of Network Enablers (NEs) in a Kubernetes cluster is different from a standard deployment.
- Deploying Cloud Native CAMEL Gateway
  When deploying CAMEL Gateway as part of the reference architecture, you can use any namespace within the Kubernetes clusters. You can make multiple deployments into multiple namespaces, for example, to have different environments for testing and production.
- Enabling MEF Publishing
  An SSH key, used for password-less SSH login, is required to enable _glossary/mef.html publishing in Kubernetes. A secret holds the SSH private key, and the key is specified in Helm values. The SSH key allows administrators to write to a target directory in a secure manner without having to enter a passphrase.
- Encrypted Interface Engine Replay Overview
  Transport-layer security (TLS) encryption can be enabled for replay between processing and publishing pods and between engines.
- MATRIXX Reference for MongoDB
  Cloud native MATRIXX deployments support MongoDB as an event store using the MongoDB Enterprise Operator for Kubernetes.
- MATRIXX Reference for Apache Kafka
  The Event Streaming Framework provides a connector for sending event stream data to Apache Kafka. Using 5G event streaming also requires Apache Kafka, which is not provided with MATRIXX. You can use a Helm chart to install and configure Apache Kafka.
- MATRIXX Reference for Ingress
  A Kubernetes Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.
- Debugging and Failure Recovery
  MATRIXX components store log files in persistent storage on the node by default, in a persistent volume (PV) at the location specified in global.storage.localStorageDir property (/home/data by default).
- Upgrading Engine Operator-Based MATRIXX Installations
  Upgrading cloud native MATRIXX has several stages.
- Migrating to Topology Operator
  This example shows migration from a MATRIXX version XXXX deployment managed by Engine Operator to a deployment managed by Topology Operator. The same approach applies when migrating from Engine Controller.
- Migration from a Bare Metal to a Cloud Native Deployment
  Migrating from a bare metal MATRIXX deployment to a cloud native deployment (from the same MATRIXX version) requires temporarily using a hybrid platform made up of components from both deployments. This hybrid environment allows migration to the cloud native deployment without disruption of services.
- The Admin Service
  Operations teams can use the Admin Service to administer a MATRIXX deployment with domain-specific commands based on the components that are installed. It allows administration of the installation without providing unrestricted direct access to the Kubernetes cluster. You can secure commands based on user roles and audit command usage.
- Uninstall MATRIXX Engine with Helm
  Use the helm uninstall command to uninstall MATRIXX Engine components installed from the Helm chart.
- Appendixes
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Removing a Sub-Domain

When removing a sub-domain, upgrading the master namespace must be done in two steps.

First, perform an upgrade using topology-additional-subdomain.yaml but with all engines in the sub-domain to be removed (s2) configured with the ignore property set to true:

helm --kube-context context1 upgrade mtx-operators matrixx/matrixx -n matrixx-operators -f base.yaml -f topology-additional-subdomain.yaml -f cluster1.yaml --set global.topology.domains[0].subdomains[1].engines[0].ignore=true --set global.topology.domains[0].subdomains[1].engines[1].ignore=true --version matrixx_version

This stops and then deletes the s2e1 and s2e2 engines. Next, upgrade the master namespace again using the topology-initial.yaml file to perform final cleanup:

helm --kube-context context1 upgrade mtx-operators matrixx/matrixx -n matrixx-operators -f base.yaml -f topology-initial.yaml -f cluster1.yaml --version matrixx_version

Note: If you attempt to upgrade the master namespace in one step, the subdomain-operator-s2, engine-operator-s2e1, and engine-operator-s2e2 pods are deleted before they are able to finish cleaning up the sub-domain 2 engines.

Upgrade and uninstall the agent and engine namespaces with the following commands:

helm --kube-context context2 upgrade mtx-operators matrixx/matrixx -n matrixx-operators -f base.yaml  -f topology-initial.yaml -f cluster2.yaml --version matrixx_version
helm --kube-context context1 upgrade mtx-engine-s1 matrixx/matrixx -n matrixx-engine-s1 -f base.yaml  -f topology-initial.yaml -f cluster1.yaml --version matrixx_version
helm --kube-context context2 upgrade mtx-engine-s1 matrixx/matrixx -n matrixx-engine-s1 -f base.yaml  -f topology-initial.yaml -f cluster2.yaml --version matrixx_version
helm --kube-context context1 uninstall mtx-engine-s2 -n matrixx-engine-s2
helm --kube-context context2 uninstall mtx-engine-s2 -n matrixx-engine-s2
kubectl --context context1 delete ns matrixx-engine-s2
kubectl --context context2 delete ns matrixx-engine-s2

Finally, upgrade the Traffic Routing Agent (TRA) namespace with the following command:

helm --kube-context context1 upgrade mtx-tra matrixx/matrixx -n matrixx-tra -f base.yaml -f topology-initial.yaml -f cluster1.yaml -f tra.yaml --version matrixx_version