Home
MATRIXX Installation and Upgrade
Topology Operator
The Topology Operator method of MATRIXX Engine pod management utilizes multiple custom resources (CRs) and operator pods working together to manage the installation and upgrade of multiple sub-domains and engines across Kubernetes clusters, including the loading of pricing and taxation data.
Health Checks
The subdomain-health-checker-sX and engine-health-checker-sXeY pods run multiple containers, each of which is a separate health check.
Cluster Monitor
Cluster Monitor, configured with the domain name and HTTP server port of Cluster Manager, collects node and cluster data. It analyzes the data and shuts down nodes as needed.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
- About MATRIXX Installation and Upgrade
  MATRIXX Installation and Upgrade describes the tasks required to install MATRIXX components as containers using Kubernetes and Helm.
- Cloud Native Infrastructure Requirements
  Cloud native MATRIXX deployments have infrastructure requirements for third-party software versions, Kubernetes pod characteristics, and container characteristics. Nodes running MATRIXX components have operating system, memory, networking, and storage requirements.
- Topology Operator and Engine Operator Feature Differences
  Different MATRIXX features are supported depending on whether your installation is based on Topology Operator or Engine Operator.
- Installing MATRIXX
  A cloud native MATRIXX installation consists of containers installed with the MATRIXX Helm chart in a multi-node Kubernetes cluster (or clusters), requiring MATRIXX images and MATRIXX Engine custom resource definitions and controllers.
- Topology Operator Installation and Upgrade Examples
  These examples show how you can configure the installation of multiple MATRIXX Engines in a single namespace, multiple namespaces in a single cluster, and multiple clusters.
- Configuring MATRIXX
  MATRIXX is configured using Helm properties. Questions and answers in create_config.info files can also provide configuration using configuration sources. For information about how to configure MATRIXX, see the discussions of those topics in MATRIXX Configuration.
- Topology Operator
  The Topology Operator method of MATRIXX Engine pod management utilizes multiple custom resources (CRs) and operator pods working together to manage the installation and upgrade of multiple sub-domains and engines across Kubernetes clusters, including the loading of pricing and taxation data.
  - Adding or Removing an Engine or Sub-Domain
    Adding a MATRIXX Engine or sub-domain is similar to the processes used for configuration, maintenance, or software upgrades. This should be performed in the following order:
  - Health Checks
    The subdomain-health-checker-sX and engine-health-checker-sXeY pods run multiple containers, each of which is a separate health check.
    - Sub-Domain Health Checks
      The sub-domain health checkers are deployed per sub-domain and are responsible for checking MATRIXX Engine health within the sub-domain. Two of the sub-domain health checker components are the gtc-sync-health-check and subdomain-health-check-brain containers.
    - Cluster Monitor
      Cluster Monitor, configured with the domain name and HTTP server port of Cluster Manager, collects node and cluster data. It analyzes the data and shuts down nodes as needed.
    - Engine Monitoring
      MATRIXX Engines are detected by requesting the engine custom resource (CR) files using topology-agent. The files contain the engine details including status and auto-healing failure count.
  - Disaster Recovery
    In multi-cluster installations, if a cluster fails, Topology Operator recovers in different ways depending on whether the cluster contained master instances or not.
  - Changes to Custom Resource Definitions
    MtxSubdomain and MtxEngine custom resource definitions (CRDs) for Pricing Controller, Engine Controller, and Engine Operator-based deployments are in the matrixx.com group. MtxTopology, MtxSubdomain, and MtxEngine CRDs for use with Topology Operator-based deployments are in the matrixx.matrixx.com group.
  - Manually Starting and Stopping Engines
    Do not use the stop_engine and start_engine scripts with Topology Operator-based deployments. Instead, to start or stop MATRIXX Engine, patch the MtxEngine CR. Start and stop actions are then handled by the operators.
  - Scaling MATRIXX Engine Pods with Helm
    In Topology Operator-based installations, increase or decrease the MATRIXX Engine pod replica count with the helm upgrade command, in the same way as any other configuration change.
  - Reconciliation Errors
    Sometimes errors may be seen in the logs of the various different operators as they try to reconcile the current state and target state. In certain cases these errors can be ignored.
  - Uninstalling a Topology Operator-Based Deployment
    To uninstall a Topology Operator-based deployment, uninstall the release, then delete the namespace.
  - Topology Operator Configuration
    Configure MATRIXX deployments based on Topology Operator by adding values for properties from the MATRIXX Helm chart to your Helm values files.
  - Topology Operator Pod Annotations
    Topology Operator components have pod annotations available to match, including annotations added to preexisting MATRIXX.
- Scaling MATRIXX
  Use Helm and Kubernetes to manually add or remove pods as necessary. You can also use Kubernetes to manage resources available to MATRIXX Engine pods.
- Network Enablers in a Kubernetes Cluster
  For Call Control Framework (CCF), only the deployment of Network Enablers (NEs) in a Kubernetes cluster is different from a standard deployment.
- Deploying Cloud Native CAMEL Gateway
  When deploying CAMEL Gateway as part of the reference architecture, you can use any namespace within the Kubernetes clusters. You can make multiple deployments into multiple namespaces, for example, to have different environments for testing and production.
- Enabling MEF Publishing
  An SSH key, used for password-less SSH login, is required to enable _glossary/mef.html publishing in Kubernetes. A secret holds the SSH private key, and the key is specified in Helm values. The SSH key allows administrators to write to a target directory in a secure manner without having to enter a passphrase.
- Encrypted Interface Engine Replay Overview
  Transport-layer security (TLS) encryption can be enabled for replay between processing and publishing pods and between engines.
- MATRIXX Reference for MongoDB
  Cloud native MATRIXX deployments support MongoDB as an event store using the MongoDB Enterprise Operator for Kubernetes.
- MATRIXX Reference for Apache Kafka
  The Event Streaming Framework provides a connector for sending event stream data to Apache Kafka. Using 5G event streaming also requires Apache Kafka, which is not provided with MATRIXX. You can use a Helm chart to install and configure Apache Kafka.
- MATRIXX Reference for Ingress
  A Kubernetes Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.
- Debugging and Failure Recovery
  MATRIXX components store log files in persistent storage on the node by default, in a persistent volume (PV) at the location specified in global.storage.localStorageDir property (/home/data by default).
- Upgrading Engine Operator-Based MATRIXX Installations
  Upgrading cloud native MATRIXX has several stages.
- Migrating to Topology Operator
  This example shows migration from a MATRIXX version XXXX deployment managed by Engine Operator to a deployment managed by Topology Operator. The same approach applies when migrating from Engine Controller.
- Migration from a Bare Metal to a Cloud Native Deployment
  Migrating from a bare metal MATRIXX deployment to a cloud native deployment (from the same MATRIXX version) requires temporarily using a hybrid platform made up of components from both deployments. This hybrid environment allows migration to the cloud native deployment without disruption of services.
- The Admin Service
  Operations teams can use the Admin Service to administer a MATRIXX deployment with domain-specific commands based on the components that are installed. It allows administration of the installation without providing unrestricted direct access to the Kubernetes cluster. You can secure commands based on user roles and audit command usage.
- Uninstall MATRIXX Engine with Helm
  Use the helm uninstall command to uninstall MATRIXX Engine components installed from the Helm chart.
- Appendixes
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Cluster Monitor

Cluster Monitor, configured with the domain name and HTTP server port of Cluster Manager, collects node and cluster data. It analyzes the data and shuts down nodes as needed.

Cluster Monitor gets the node status of each node, waiting until each node state becomes active as necessary. Cluster Monitor compares cluster manager protocol (CMP) cluster IDs from each node. A split-brain condition is detected if CMP cluster IDs are not the same. This is similar to how Traffic Routing Agent (TRA) detects split brain conditions.

If a split-brain condition is detected, Cluster Monitor shuts down the node based on the following criteria:

If the number of nodes in each cluster are different, then the cluster with the lower number of nodes is shut down.
If each cluster has the same number of nodes, then the cluster that has the largest CMP cluster ID is shut down, as that is the cluster that formed most recently.