Home
MATRIXX Monitoring and Logging
Metrics and Alerts
The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
Monitoring the MATRIXX Engine Environment Locally
The following topics explain how to monitor the MATRIXX Engine locally using the internal MATRIXX Engine tools.
Monitoring Recommendations
The following information includes recommended actions for monitoring a real-time MATRIXX production system. Operating system and hardware monitoring should also be implemented but is not described in this document.
MATRIXX Proxy Processes
Monitor proxy processes before starting the engine to confirm that the Apache ActiveMQ and Notification Server processes are running before MATRIXX Engine starts. This ensures that no notifications are lost when the engine comes online.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
- About MATRIXX Monitoring and Logging
  MATRIXX Monitoring and Logging describes monitoring and logging for MATRIXX components in a MATRIXX environment.
- Metrics and Alerts
  The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
  - Monitoring with Prometheus and Grafana
    The following information describes how to implement monitoring through Prometheus in a containerized environment for MATRIXX. Information collected by Prometheus can be displayed with tools such as Grafana.
  - Cluster HA States
    A cluster can change high availability (HA) states during engine start up, manual switchover, failover, and when the connection to a remote peer cluster is lost.
  - MATRIXX Engine Start-Up Sequence
    A Topology Operator engine-starter instance starts MATRIXX Engine and engine-level Traffic Routing Agent (TRA-PROC and TRA-PUB) pods.
  - Monitoring the MATRIXX Engine Environment Locally
    The following topics explain how to monitor the MATRIXX Engine locally using the internal MATRIXX Engine tools.
    - Server States
      Servers can have several states that indicate the stage they are in during processing. They can be waiting to synchronize databases, synchronizing databases, actively processing transactions, preparing to stop, or stopping.
    - Cluster HA States
      A cluster can change high availability (HA) states during engine start up, manual switchover, failover, and when the connection to a remote peer cluster is lost.
    - MATRIXX SNMP Notifications
      MATRIXX uses trap-directed notifications to alert your SNMPv2 or SNMPv3 Network Operations Center (NOC) when a change occurs in a service status, an error occurs, a threshold has been crossed, or the system topology changes. It does this by sending a trap of the event.
    - Configure and Enable System Monitor
      You configure the optional MATRIXX System Monitor by editing the sysmon_config.xml file. After the file is configured, System Monitor continuously monitors the current MATRIXX Engine processing server.
    - Monitoring Recommendations
      The following information includes recommended actions for monitoring a real-time MATRIXX production system. Operating system and hardware monitoring should also be implemented but is not described in this document.
      - MATRIXX Engine Processes
        Monitor engine processes when you start an engine to confirm the engine processes are initiated and monitor them when you stop an engine to confirm the engine processes are terminated.
      - Monitoring Automatically-Captured Diagnostic Statistics
        By default each MATRIXX Engine runs the capture_diagstats_tofile.py script every five minutes to capture diagnostic statistics in a ${MTX-LOG-DIR}/snmp_engine_id-stats.log file. Also by default, this program closes that log file and opens a new one to take its place when it reaches a configurable time or size limit. The information in these files is valuable for debugging problems.
      - Analyzing Diagnostic Statistics
        Use the scripts in this section to read and analyze captured diagnostic statistics, plot the collected data by specified Management Information Base (MIB) files and time range, and print out the results.
      - MATRIXX Proxy Processes
        Monitor proxy processes before starting the engine to confirm that the Apache ActiveMQ and Notification Server processes are running before MATRIXX Engine starts. This ensures that no notifications are lost when the engine comes online.
      - File Systems
        File systems should be monitored for details such as local SSD free space and gaps in the file sequence.
      - MATRIXX Log Files
        Depending on the MATRIXX Engine configuration, you can view MATRIXX log files to gather information about runtime processing.
      - Reading Trace Logs
        The tables in trace log files contain information about data fields as well as their values.
    - Monitor Database Sync Progress
      You can monitor the database sync operation of a cluster during engine start up, engine failover, and server start up.
    - Monitor Transaction Replay Progress on the Standby Engine
      Run the print_blade_stats.py script with the -R option to display the number of checkpoint log files during local transaction replay and the number of outstanding replay batches during remote replay that are outstanding when completing an InitDatabase request.
    - Monitor the Availability of the STANDBY Cluster
      The STANDBY cluster responds to Diameter Device-Watchdog-Request (DWR) messages to let the network application know it is running so it can be ready to take over processing if a failover occurs. You can monitor the sending and receiving of DW messages by monitoring SNMP notifications and by running the print_blade_stats.py script.
    - Display Memory Statistics
      There are two types of memory statistics tracked in MATRIXX Engine: those related to a specific database and the total system memory allocated and in-use.
  - Custom System Monitor Configuration
    You configure the optional MATRIXX System Monitor by editing the sysmon_config.xml file. After the file is configured, System Monitor continuously monitors the MATRIXX Engine processing pod.
  - Alert Management Integration with SNMP Traps
    In a Cloud Native environment you can integrate the statistics and metrics data collected by Prometheus with other systems, such as Alert Manager, and you can configure snmp-notifier to convert the generated alerts to SNMP traps for reporting in external monitoring systems.
  - Retrieving Core Dump Information
    Core dumps are files that include the complete contents of the memory of a process at a point in time. Core dump handling in MATRIXX must be enabled per namespace. By default this feature is disabled and the commonly mounted /coredumps directory is empty until this feature is enabled.
- Logging
  The following describes logging information for MATRIXX.
- Tracing
  The following information describes tracing information for MATRIXX.
- Monitoring Components
  The following information describes monitoring different components for MATRIXX.
- Appendixes
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

MATRIXX Proxy Processes

Monitor proxy processes before starting the engine to confirm that the Apache ActiveMQ and Notification Server processes are running before MATRIXX Engine starts. This ensures that no notifications are lost when the engine comes online.

MATRIXX proxy processes include:

Gateway Proxy
Apache ActiveMQ
MATRIXX ActiveMQ Notification Server

For more information about the MATRIXX environment variables, see the discussion about installation directories, navigation shortcuts, and environment variables in MATRIXX Installation and Upgrade.

Gateway Proxy

The following sample shows Gateway Proxy process details.

mtx       7172     1  0 02:34 ?        00:00:42 java 
-DMTX_LOG_DIR=/var/log/mtx 
-Dlog4j.configurationFile=file:///opt/mtx/conf/log4j2-mtx_gateway_proxy.xml
-jar /opt/mtx/bin/gateway_proxy.jar 
-props:gateway_proxy.yaml &

Apache ActiveMQ

The following sample shows Apache ActiveMQ process details.

activemq  6525     1  1 04:50 ?        00:00:16 /usr/bin/java -Xms1G -Xmx1G 
-Djava.util.logging.config.file=logging.properties 
-Dhawtio.realm=activemq 
-Dhawtio.role=admins 
-Dhawtio.rolePrincipalClasses=org.apache.activemq.jaas.GroupPrincipal 
-Djava.security.auth.login.config=/opt/apache-activemq-5.9.0/conf/login.config 
-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote 
-Djava.awt.headless=true 
-Djava.io.tmpdir=/opt/apache-activemq-5.9.0/tmp 
-Dactivemq.classpath=/opt/apache-activemq-5.9.0/conf; 
-Dactivemq.home=/opt/apache-activemq-5.9.0 
-Dactivemq.base=/opt/apache-activemq-5.9.0 
-Dactivemq.conf=/opt/apache-activemq-5.9.0/conf 
-Dactivemq.data=/opt/apache-activemq-5.9.0/data 
-jar /opt/apache-activemq-5.9.0/bin/activemq.jar start

MATRIXX Notification Server

The following sample shows the Notification Server process details.

mtx      31747     1 31 11:29 pts/0    00:00:15 java 
-DMTX_LOG_DIR=/var/log/mtx 
-Dlog4j.debug=true 
-Dlog4j.configurationFile=file:///opt/mtx/conf/log4j2-mtx_notifier_camel.xml
-Dloader.debug=true 
-Dloader.config.location=/opt/mtx/conf/application_mtx_notifier_camel.properties 
-jar mtx_notifier_camel.jar

For more information about proxy processes, see the discussion about MATRIXX gateways and web apps in MATRIXX Installation and Upgrade.