Home
MATRIXX Installation and Upgrade
Topology Operator
The Topology Operator method of MATRIXX Engine pod management utilizes multiple custom resources (CRs) and operator pods working together to manage the installation and upgrade of multiple sub-domains and engines across Kubernetes clusters, including the loading of pricing and taxation data.
Health Checks
The subdomain-health-checker-sX and engine-health-checker-sXeY pods run multiple containers, each of which is a separate health check.
Sub-Domain Health Checks
The sub-domain health checkers are deployed per sub-domain and are responsible for checking MATRIXX Engine health within the sub-domain. Two of the sub-domain health checker components are the gtc-sync-health-check and subdomain-health-check-brain containers.
GTC Sync Health Check
The gtc-sync-health-check container continuously checks the GTC replay between engines in the sub-domain, and between the processing and publishing clusters of each MATRIXX Engine.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
- About MATRIXX Installation and Upgrade
  MATRIXX Installation and Upgrade describes the tasks required to install MATRIXX components as containers using Kubernetes and Helm.
- Cloud Native Infrastructure Requirements
  Cloud native MATRIXX deployments have infrastructure requirements for third-party software versions, Kubernetes pod characteristics, and container characteristics. Nodes running MATRIXX components have operating system, memory, networking, and storage requirements.
- Topology Operator and Engine Operator Feature Differences
  Different MATRIXX features are supported depending on whether your installation is based on Topology Operator or Engine Operator.
- Installing MATRIXX
  A cloud native MATRIXX installation consists of containers installed with the MATRIXX Helm chart in a multi-node Kubernetes cluster (or clusters), requiring MATRIXX images and MATRIXX Engine custom resource definitions and controllers.
- Topology Operator Installation and Upgrade Examples
  These examples show how you can configure the installation of multiple MATRIXX Engines in a single namespace, multiple namespaces in a single cluster, and multiple clusters.
- Configuring MATRIXX
  MATRIXX is configured using Helm properties. Questions and answers in create_config.info files can also provide configuration using configuration sources. For information about how to configure MATRIXX, see the discussions of those topics in MATRIXX Configuration.
- Topology Operator
  The Topology Operator method of MATRIXX Engine pod management utilizes multiple custom resources (CRs) and operator pods working together to manage the installation and upgrade of multiple sub-domains and engines across Kubernetes clusters, including the loading of pricing and taxation data.
  - Adding or Removing an Engine or Sub-Domain
    Adding a MATRIXX Engine or sub-domain is similar to the processes used for configuration, maintenance, or software upgrades. This should be performed in the following order:
  - Health Checks
    The subdomain-health-checker-sX and engine-health-checker-sXeY pods run multiple containers, each of which is a separate health check.
    - Sub-Domain Health Checks
      The sub-domain health checkers are deployed per sub-domain and are responsible for checking MATRIXX Engine health within the sub-domain. Two of the sub-domain health checker components are the gtc-sync-health-check and subdomain-health-check-brain containers.
      - GTC Sync Health Check
        The gtc-sync-health-check container continuously checks the GTC replay between engines in the sub-domain, and between the processing and publishing clusters of each MATRIXX Engine.
      - Sub-Domain Health Check Brain Recovery Operations
        The subdomain-health-check-brain attempts recovery from four conditions where a failed condition prevents considering the next decision.
    - Cluster Monitor
      Cluster Monitor, configured with the domain name and HTTP server port of Cluster Manager, collects node and cluster data. It analyzes the data and shuts down nodes as needed.
    - Engine Monitoring
      MATRIXX Engines are detected by requesting the engine custom resource (CR) files using topology-agent. The files contain the engine details including status and auto-healing failure count.
  - Disaster Recovery
    In multi-cluster installations, if a cluster fails, Topology Operator recovers in different ways depending on whether the cluster contained master instances or not.
  - Changes to Custom Resource Definitions
    MtxSubdomain and MtxEngine custom resource definitions (CRDs) for Pricing Controller, Engine Controller, and Engine Operator-based deployments are in the matrixx.com group. MtxTopology, MtxSubdomain, and MtxEngine CRDs for use with Topology Operator-based deployments are in the matrixx.matrixx.com group.
  - Manually Starting and Stopping Engines
    Do not use the stop_engine and start_engine scripts with Topology Operator-based deployments. Instead, to start or stop MATRIXX Engine, patch the MtxEngine CR. Start and stop actions are then handled by the operators.
  - Scaling MATRIXX Engine Pods with Helm
    In Topology Operator-based installations, increase or decrease the MATRIXX Engine pod replica count with the helm upgrade command, in the same way as any other configuration change.
  - Reconciliation Errors
    Sometimes errors may be seen in the logs of the various different operators as they try to reconcile the current state and target state. In certain cases these errors can be ignored.
  - Uninstalling a Topology Operator-Based Deployment
    To uninstall a Topology Operator-based deployment, uninstall the release, then delete the namespace.
  - Topology Operator Configuration
    Configure MATRIXX deployments based on Topology Operator by adding values for properties from the MATRIXX Helm chart to your Helm values files.
  - Topology Operator Pod Annotations
    Topology Operator components have pod annotations available to match, including annotations added to preexisting MATRIXX.
- Scaling MATRIXX
  Use Helm and Kubernetes to manually add or remove pods as necessary. You can also use Kubernetes to manage resources available to MATRIXX Engine pods.
- Network Enablers in a Kubernetes Cluster
  For Call Control Framework (CCF), only the deployment of Network Enablers (NEs) in a Kubernetes cluster is different from a standard deployment.
- Deploying Cloud Native CAMEL Gateway
  When deploying CAMEL Gateway as part of the reference architecture, you can use any namespace within the Kubernetes clusters. You can make multiple deployments into multiple namespaces, for example, to have different environments for testing and production.
- Enabling MEF Publishing
  An SSH key, used for password-less SSH login, is required to enable _glossary/mef.html publishing in Kubernetes. A secret holds the SSH private key, and the key is specified in Helm values. The SSH key allows administrators to write to a target directory in a secure manner without having to enter a passphrase.
- Encrypted Interface Engine Replay Overview
  Transport-layer security (TLS) encryption can be enabled for replay between processing and publishing pods and between engines.
- MATRIXX Reference for MongoDB
  Cloud native MATRIXX deployments support MongoDB as an event store using the MongoDB Enterprise Operator for Kubernetes.
- MATRIXX Reference for Apache Kafka
  The Event Streaming Framework provides a connector for sending event stream data to Apache Kafka. Using 5G event streaming also requires Apache Kafka, which is not provided with MATRIXX. You can use a Helm chart to install and configure Apache Kafka.
- MATRIXX Reference for Ingress
  A Kubernetes Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.
- Debugging and Failure Recovery
  MATRIXX components store log files in persistent storage on the node by default, in a persistent volume (PV) at the location specified in global.storage.localStorageDir property (/home/data by default).
- Upgrading Engine Operator-Based MATRIXX Installations
  Upgrading cloud native MATRIXX has several stages.
- Migrating to Topology Operator
  This example shows migration from a MATRIXX version XXXX deployment managed by Engine Operator to a deployment managed by Topology Operator. The same approach applies when migrating from Engine Controller.
- Migration from a Bare Metal to a Cloud Native Deployment
  Migrating from a bare metal MATRIXX deployment to a cloud native deployment (from the same MATRIXX version) requires temporarily using a hybrid platform made up of components from both deployments. This hybrid environment allows migration to the cloud native deployment without disruption of services.
- The Admin Service
  Operations teams can use the Admin Service to administer a MATRIXX deployment with domain-specific commands based on the components that are installed. It allows administration of the installation without providing unrestricted direct access to the Kubernetes cluster. You can secure commands based on user roles and audit command usage.
- Uninstall MATRIXX Engine with Helm
  Use the helm uninstall command to uninstall MATRIXX Engine components installed from the Helm chart.
- Appendixes
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

GTC Sync Health Check

The gtc-sync-health-check container continuously checks the GTC replay between engines in the sub-domain, and between the processing and publishing clusters of each MATRIXX Engine.

A GTC out-of-sync is not detected unless one of the following conditions is met:

All engines have been started.
An active engine is running with any other engines disabled after the maximum configured number of attempts to auto-heal.

The GTC sync health check reports an error if either of the following conditions arise:

A constant GTC value for the configured period of time, in which case all values detected during the period must be identical, nonzero, and at least one of the current GTC and last replayed GTC must be static.
A GTC value exceeding the configured maximum value which does not decrease.

Multiple errors can be detected during the same period of time, for example in different engines. The errors are reported to the sub-domain health check brain container for it to make a decision on what action to take, if any.

Having established that an active engine and standby engine are present, and that the transactions per second are not zero, a five-minute cycle of GTC monitoring is performed. (All time intervals are configurable.) The following GTC detections are performed, where engine 1 is the active engine and engine 2 is the standby engine.

When one engine is available:

Engine 1 processing — engine 1 publishing

When two engines are available:

Engine 1 processing — engine 2 processing
Engine 1 processing — engine 1 publishing
Engine 2 processing — engine 2 publishing

The current GTC to last-replay GTC gap between the pods are checked every 10 seconds, for 5 minutes. A gap exceeding a configured maximum value during the detection period, that does not decrease, indicates an out-of-sync error. A gap that does not change for the entire detection where either the current GTC or last replay GTC is not moving also indicates an out-of-sync error.

The two components of the health check communicate using a state file created and updated by the gtc-sync-health-check container. This gtc_sync_state file is stored in shared logging storage. If there is no persistent storage enabled then the state file is saved within the sub-domain health checker pod. The file is empty if there have been no GTC out-of-sync detections.