Home
MATRIXX Monitoring and Logging
Metrics and Alerts
The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
Monitoring the MATRIXX Engine Environment Locally
The following topics explain how to monitor the MATRIXX Engine locally using the internal MATRIXX Engine tools.
Monitor Database Sync Progress
You can monitor the database sync operation of a cluster during engine start up, engine failover, and server start up.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
- About MATRIXX Monitoring and Logging
  MATRIXX Monitoring and Logging describes monitoring and logging for MATRIXX components in a MATRIXX environment.
- Metrics and Alerts
  The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
  - Monitoring with Prometheus and Grafana
    The following information describes how to implement monitoring through Prometheus in a containerized environment for MATRIXX. Information collected by Prometheus can be displayed with tools such as Grafana.
  - Cluster HA States
    A cluster can change high availability (HA) states during engine start up, manual switchover, failover, and when the connection to a remote peer cluster is lost.
  - MATRIXX Engine Start-Up Sequence
    A Topology Operator engine-starter instance starts MATRIXX Engine and engine-level Traffic Routing Agent (TRA-PROC and TRA-PUB) pods.
  - Monitoring the MATRIXX Engine Environment Locally
    The following topics explain how to monitor the MATRIXX Engine locally using the internal MATRIXX Engine tools.
    - Server States
      Servers can have several states that indicate the stage they are in during processing. They can be waiting to synchronize databases, synchronizing databases, actively processing transactions, preparing to stop, or stopping.
    - Cluster HA States
      A cluster can change high availability (HA) states during engine start up, manual switchover, failover, and when the connection to a remote peer cluster is lost.
    - MATRIXX SNMP Notifications
      MATRIXX uses trap-directed notifications to alert your SNMPv2 or SNMPv3 Network Operations Center (NOC) when a change occurs in a service status, an error occurs, a threshold has been crossed, or the system topology changes. It does this by sending a trap of the event.
    - Configure and Enable System Monitor
      You configure the optional MATRIXX System Monitor by editing the sysmon_config.xml file. After the file is configured, System Monitor continuously monitors the current MATRIXX Engine processing server.
    - Monitoring Recommendations
      The following information includes recommended actions for monitoring a real-time MATRIXX production system. Operating system and hardware monitoring should also be implemented but is not described in this document.
    - Monitor Database Sync Progress
      You can monitor the database sync operation of a cluster during engine start up, engine failover, and server start up.
      - Example — Processing Pod: Database Sync Statistics
        This sample output from the print_blade_stats.py -C command shows statistics as a processing pod synchronizes its databases and then becomes active.
    - Monitor Transaction Replay Progress on the Standby Engine
      Run the print_blade_stats.py script with the -R option to display the number of checkpoint log files during local transaction replay and the number of outstanding replay batches during remote replay that are outstanding when completing an InitDatabase request.
    - Monitor the Availability of the STANDBY Cluster
      The STANDBY cluster responds to Diameter Device-Watchdog-Request (DWR) messages to let the network application know it is running so it can be ready to take over processing if a failover occurs. You can monitor the sending and receiving of DW messages by monitoring SNMP notifications and by running the print_blade_stats.py script.
    - Display Memory Statistics
      There are two types of memory statistics tracked in MATRIXX Engine: those related to a specific database and the total system memory allocated and in-use.
  - Custom System Monitor Configuration
    You configure the optional MATRIXX System Monitor by editing the sysmon_config.xml file. After the file is configured, System Monitor continuously monitors the MATRIXX Engine processing pod.
  - Alert Management Integration with SNMP Traps
    In a Cloud Native environment you can integrate the statistics and metrics data collected by Prometheus with other systems, such as Alert Manager, and you can configure snmp-notifier to convert the generated alerts to SNMP traps for reporting in external monitoring systems.
  - Retrieving Core Dump Information
    Core dumps are files that include the complete contents of the memory of a process at a point in time. Core dump handling in MATRIXX must be enabled per namespace. By default this feature is disabled and the commonly mounted /coredumps directory is empty until this feature is enabled.
- Logging
  The following describes logging information for MATRIXX.
- Tracing
  The following information describes tracing information for MATRIXX.
- Monitoring Components
  The following information describes monitoring different components for MATRIXX.
- Appendixes
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Monitor Database Sync Progress

You can monitor the database sync operation of a cluster during engine start up, engine failover, and server start up.

About this task

When a standby cluster starts up, only one server at a time is allowed to join the transaction topology and synchronize its databases from the active cluster. During the time the server is updating its database, it has a Server State of sync. After its databases are up-to-date, the Server State changes to active. When a standby cluster switches to the active cluster during a failover operation, all servers have a Server State of sync while processing outstanding transactions from the active cluster. When finished, Server State changes to active.

Perform this task on a server that is in the cluster that is updating its databases.

Procedure

Enter the following command and replace seconds with the number of seconds to reprint the statistics.

print_blade_stats.py -C -r seconds