Home
MATRIXX Monitoring and Logging
Metrics and Alerts
The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
Monitoring the MATRIXX Engine Environment Locally
The following topics explain how to monitor the MATRIXX Engine locally using the internal MATRIXX Engine tools.
Server States
Servers can have several states that indicate the stage they are in during processing. They can be waiting to synchronize databases, synchronizing databases, actively processing transactions, preparing to stop, or stopping.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
- About MATRIXX Monitoring and Logging
  MATRIXX Monitoring and Logging describes monitoring and logging for MATRIXX components in a MATRIXX environment.
- Metrics and Alerts
  The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
  - Monitoring with Prometheus and Grafana
    The following information describes how to implement monitoring through Prometheus in a containerized environment for MATRIXX. Information collected by Prometheus can be displayed with tools such as Grafana.
  - Cluster HA States
    A cluster can change high availability (HA) states during engine start up, manual switchover, failover, and when the connection to a remote peer cluster is lost.
  - MATRIXX Engine Start-Up Sequence
    A Topology Operator engine-starter instance starts MATRIXX Engine and engine-level Traffic Routing Agent (TRA-PROC and TRA-PUB) pods.
  - Monitoring the MATRIXX Engine Environment Locally
    The following topics explain how to monitor the MATRIXX Engine locally using the internal MATRIXX Engine tools.
    - Server States
      Servers can have several states that indicate the stage they are in during processing. They can be waiting to synchronize databases, synchronizing databases, actively processing transactions, preparing to stop, or stopping.
      - Display Server HA States
        Perform this task to view the current role and high availability (HA) state of the MATRIXX servers in a cluster and the HA state of any peer clusters. The service state changes during engine start up and failover operations to indicate whether it is ready for processing or synchronizing its databases.
      - Display All Server Statistics
        You can view runtime statistics for a server, including database statistics, Diameter statistics, MATRIXX service statistics, queue statistics, transaction replay statistics, and memory statistics.
      - Reset Server Statistics
        You can reset the SNMP statistics for the Diameter Gateway and the MDC Gateway. You can specify the service using the command options or you can specify the service abbreviations.
    - Cluster HA States
      A cluster can change high availability (HA) states during engine start up, manual switchover, failover, and when the connection to a remote peer cluster is lost.
    - MATRIXX SNMP Notifications
      MATRIXX uses trap-directed notifications to alert your SNMPv2 or SNMPv3 Network Operations Center (NOC) when a change occurs in a service status, an error occurs, a threshold has been crossed, or the system topology changes. It does this by sending a trap of the event.
    - Configure and Enable System Monitor
      You configure the optional MATRIXX System Monitor by editing the sysmon_config.xml file. After the file is configured, System Monitor continuously monitors the current MATRIXX Engine processing server.
    - Monitoring Recommendations
      The following information includes recommended actions for monitoring a real-time MATRIXX production system. Operating system and hardware monitoring should also be implemented but is not described in this document.
    - Monitor Database Sync Progress
      You can monitor the database sync operation of a cluster during engine start up, engine failover, and server start up.
    - Monitor Transaction Replay Progress on the Standby Engine
      Run the print_blade_stats.py script with the -R option to display the number of checkpoint log files during local transaction replay and the number of outstanding replay batches during remote replay that are outstanding when completing an InitDatabase request.
    - Monitor the Availability of the STANDBY Cluster
      The STANDBY cluster responds to Diameter Device-Watchdog-Request (DWR) messages to let the network application know it is running so it can be ready to take over processing if a failover occurs. You can monitor the sending and receiving of DW messages by monitoring SNMP notifications and by running the print_blade_stats.py script.
    - Display Memory Statistics
      There are two types of memory statistics tracked in MATRIXX Engine: those related to a specific database and the total system memory allocated and in-use.
  - Custom System Monitor Configuration
    You configure the optional MATRIXX System Monitor by editing the sysmon_config.xml file. After the file is configured, System Monitor continuously monitors the MATRIXX Engine processing pod.
  - Alert Management Integration with SNMP Traps
    In a Cloud Native environment you can integrate the statistics and metrics data collected by Prometheus with other systems, such as Alert Manager, and you can configure snmp-notifier to convert the generated alerts to SNMP traps for reporting in external monitoring systems.
  - Retrieving Core Dump Information
    Core dumps are files that include the complete contents of the memory of a process at a point in time. Core dump handling in MATRIXX must be enabled per namespace. By default this feature is disabled and the commonly mounted /coredumps directory is empty until this feature is enabled.
- Logging
  The following describes logging information for MATRIXX.
- Tracing
  The following information describes tracing information for MATRIXX.
- Monitoring Components
  The following information describes monitoring different components for MATRIXX.
- Appendixes
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Server States

Servers can have several states that indicate the stage they are in during processing. They can be waiting to synchronize databases, synchronizing databases, actively processing transactions, preparing to stop, or stopping.

Server HA States lists the states that servers can have during runtime and failover operations. The SNMP agent running on an engine server sends a notification when a server changes from one state to another.

Table 1. Server HA States
State	Description	SNMP ID
UNKNOWN	The server is stopped.	0
INIT	The server has been started and is initializing its services.	1
SYNC-WAIT SYNC-READY	With the SYNC-READY state, SYNC-WAIT indicates that the server is getting ready to start its database initialization and synchronization. These states are used together to order the sequence of the servers doing a database synchronization one at a time, during a failover operation.	2 3
SYNC	The server is performing its DB initialization and synchronization.	4
ACTIVE	The server's databases are synchronized and up-to-date and the transaction server in the server is active and ready to process transactions.	5
QUIESCE	The external diameter traffic to the server has been paused so the server can process any remaining diameter requests and transactions during the duration of the state. A server is in QUIESCE state when its cluster is failing over to a peer cluster.	6
EXIT	The server is in the process of stopping.	7
FINAL	The server is stopped.	8
STANDBY	The server is up-to-date and ready to take over processing if necessary.	9