Home
MATRIXX Monitoring and Logging
Monitoring Components
The following information describes monitoring different components for MATRIXX.
Monitoring MATRIXX Route Cache
Monitor the Route Cache by reading watermark log messages for each Route Cache table and Route Cache SNMP statistics. The SNMP statistics are defined in the matrixx_rc_mib.txt MIB file and returned by the print_snmp_stats.py script. They include statistics on subscriber and session data, statistics on insert, fetch, erase, and iterator operations, general statistics on subscriber records, and garbage collection.
Reported Route Cache Statistics
The following Route Cache (RC) statistics are reported by the print_snmp_stats.py script.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
- About MATRIXX Monitoring and Logging
  MATRIXX Monitoring and Logging describes monitoring and logging for MATRIXX components in a MATRIXX environment.
- Metrics and Alerts
  The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
- Logging
  The following describes logging information for MATRIXX.
- Tracing
  The following information describes tracing information for MATRIXX.
- Monitoring Components
  The following information describes monitoring different components for MATRIXX.
  - Monitoring MATRIXX Business API Gateway
    Business API Gateway SNMP statistics are defined in the matrixx_web_mib.txt file and you display them using the snmp-net-utils snmpwalk command.
  - Monitoring MATRIXX Call Control Framework
    The topics in this section have information about monitoring Call Control Framework (CCF) components, including Network Enabler and CAMEL Gateway.
  - Monitoring Diameter Events
    Diameter Event Logger listens to the Diameter Gateway through a TCP connection and logs diameter application messages to a JSON file.
  - Monitoring Event Streaming
    Event streaming errors are logged in the Event Streaming Framework mtx_event_streamer.log file. Event Streaming Framework uses Log4j 2.x to log error, debugging, and informational messages. You can also monitor event streaming performance using the performance_analysis_event_server.py script. For information about the script, see the discussion about performance_analysis_event_server.py.
  - Monitoring MATRIXX Route Cache
    Monitor the Route Cache by reading watermark log messages for each Route Cache table and Route Cache SNMP statistics. The SNMP statistics are defined in the matrixx_rc_mib.txt MIB file and returned by the print_snmp_stats.py script. They include statistics on subscriber and session data, statistics on insert, fetch, erase, and iterator operations, general statistics on subscriber records, and garbage collection.
    - Route Cache Watermark Log Messages
      Route Cache watermark log messages inform system administrators about the usage level of each Route Cache table, such as the table that contains cached subscriber ID data and the table that contains cached session ID data. The number of free or available route records in each Route Cache table is tracked. If watermark messages are logged frequently, or communicate a critical state, it may be an indication that the size of the table must be increased to accommodate more route records, such as more cached subscriber identity data or more cached session ID data. Route Cache watermark log messages are logged in the mtx_debug.log file.
    - Display Route Cache Statistics
      Route Cache statistics are collected for subscriber and session data, including statistics on insert, fetch, erase, and iterator operations, and general statistics on subscriber records, and garbage collection.
    - Reported Route Cache Statistics
      The following Route Cache (RC) statistics are reported by the print_snmp_stats.py script.
  - Monitoring Payment Service Logging
    Payment service logs messages to the payment-service.log.
  - Monitoring MATRIXX SNMP Statistics
    By default, the SNMP Agent running on a MATRIXX Engine monitors the status and health of all the servers and clusters it has. It collects statistics to aid in identifying potential processing issues.
  - Monitoring MATRIXX Traffic Routing Agent
    Traffic Routing Agent SNMP statistics are defined in the matrixx_tra_mib.txt MIB file and are returned by the print_snmp_stats.py script when run on any of the Traffic Routing Agent servers. They include categories such as TCP traffic, UDP traffic, Diameter Gateway traffic, and MDC Gateway traffic.
  - Monitoring MATRIXX Web Apps
    Use the MATRIXX system logs and SNMP statistics and notifications to monitor the MATRIXX web-facing applications, such as RS Gateway, Gateway Proxy, and My MATRIXX.
  - Monitoring MATRIXX SBA Gateway
    The MATRIXX SBA Gateway SNMP statistics are defined in the matrixx_web_mib.txt file, using the snmp-net-utils snmpwalk command to display them.
  - Monitoring 5G Event Streaming
    5G event streaming can be monitored with standard Kafka and custom metrics.
  - Monitoring Kafka CDR Consumer
    Use aggregated logging and the Prometheus and Grafana third-party products to monitor Kafka CDR Consumer.
- Appendixes
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Reported Route Cache Statistics

The following Route Cache (RC) statistics are reported by the print_snmp_stats.py script.

Route Cache Controller Subscriber/Admin Subscriber Statistics

Subscriber/Admin Subscriber Mdb Insert Statistics describes the insert statistics.

Table 1. Subscriber/Admin Subscriber Mdb Insert Statistics
Statistic	Description
Call In Prog	Currently active insert operations.
Call	Insert operations since RC start.
Alloc Nrp	Allocated records during insert operation since RC start.
Key Found	Keys found during insert operation since RC start.
Val Eq Cond	Matching values found during insert operation since RC start.
Update	Records updated during insert operation since RC start.
Exchange	Records exchanged during insert operation since RC start.
New	New records inserted during insert operation since RC start.
No Memory	Free record unavailable errors during insert operation since RC start.
Exists	Record already exists errors during insert new record operation since RC start.
Not Found	Record not found errors during record update operation since RC start.
Error	Errors detected during insert operation since RC start.

Subscriber/Admin Subscriber Mdb Fetch Statistics describes the fetch statistics.

Table 2. Subscriber/Admin Subscriber Mdb Fetch Statistics
Statistic	Description
Call In Prog	Currently active fetch operations.
Call	Fetch operations since RC start.
Rec Found	Records found during fetch operations since RC start.
Not Found	Records not found during fetch operations since RC start.
Error	Errors detected during fetch operations since RC start.

Subscriber/Admin Subscriber Mdb Erase Statistics describes the erase statistics.

Table 3. Subscriber/Admin Subscriber Mdb Erase Statistics
Statistic	Description
In Progress	Currently active erase operations.
Calls	Erase operations since RC start.
Key Match	Erase with key-match operations since RC start.
Prefix Match	Erase with key prefix match operations since RC start.
All Calls	Erase all records operations since RC start.
In Hash Bckt	Erase all records in hash bucket operations since RC start.
Not Found	Records not found during erase operation since RC start.
Error	Errors detected during erase operation since RC start.

Subscriber/Admin Subscriber Mdb Iterators Statistics describes iterator statistics.

Table 4. Subscriber/Admin Subscriber Mdb Iterators Statistics
Statistic	Description
New Iter Calls	New iterator method calls since RC start.
Delete Iter Calls	Delete iterator method calls since RC start.
Process Iter Calls	Process iterator method calls since RC start.
Errors	Errors detected during iterator processing calls since RC start.
EOF	Number of EOF conditions detected during iterator processing calls since RC start.
Invld ShmId	Number of Invalid SHM ID conditions detected during iterator processing calls since RC start.
Return Value	Iterator processing returns value since RC start.
Return Ptr	Iterator processing returns pointer since RC start.
Erase	Number of erased records during iterator processing since RC start.
UnLock	Number of unlocked hash buckets during iterator processing since RC start.
Next	Number of next-record operations conducted during iterator processing since RC start.

Subscriber/Admin Subscriber Mdb Garbage Collection Statistics describes garbage collection statistics.

Table 5. Subscriber/Admin Subscriber Mdb Garbage Collection Statistics
Statistic	Description
Timeout (ms)	Current garbage collection processing timeout in milliseconds.
Loops	Garbage collection processing loop counter since RC start.
Recs Erase	Records erased during garbage collection since RC start.

Subscriber/Admin Subscriber Mdb Main Info Object Statistics describes main info object statistics.

Table 6. Subscriber/Admin Subscriber Mdb Main Info Object Statistics
Statistic	Description
Total Recs	Total number of records.
Recs Avail	Total number of available records.
Recs Avail %	Percentage of all records that are available.

Route Cache Controller Session Statistics

Session Mdb Insert Statistics describes instert statistics.

Table 7. Session Mdb Insert Statistics
Statistic	Description
Call In Prog	Currently active insert operations.
Call	Insert operations since RC start.
Alloc Nrp	Allocated records during insert operation since RC start.
Key Found	Keys found during insert operation since RC start.
Val Eq Cond	Matching values found during insert operation since RC start.
Update	Records updated during insert operation since RC start.
Exchange	Records exchanged during insert operation since RC start.
New	New records inserted during insert operation since RC start.
No Memory	Free record unavailable errors during insert operation since RC start.
Exists	Record already exists errors during insert new record operation since RC start.
Not Found	Record not found errors during record update operation since RC start.
Error	Errors detected during insert operation since RC start.

Session Mdb Fetch Statistics describes fetch statistics.

Table 8. Session Mdb Fetch Statistics
Statistic	Description
Call In Prog	Currently active fetch operations.
Call	Fetch operations since RC start.
Rec Found	Records found during fetch operations since RC start.
Not Found	Records not found during fetch operations since RC start.
Error	Errors detected during fetch operations since RC start.

Session Mdb Erase Statistics describes erase statistics.

Table 9. Session Mdb Erase Statistics
Statistic	Description
In Progress	Currently active erase operations.
Calls	Erase operations since RC start.
Key Match	Erase with key-match operations since RC start.
Prefix Match	Erase with key prefix match operations since RC start.
All Calls	Erase all records operations since RC start.
In Hash Bckt	Erase all records in hash bucket operations since RC start.
Not Found	Records not found during erase operation since RC start.
Error	Errors detected during erase operation since RC start.

Subscriber/Admin Subscriber Mdb Iterators Statistics describes iterator statistics.

Table 10. Subscriber/Admin Subscriber Mdb Iterators Statistics
Statistic	Description
New Iter Calls	New iterator method calls since RC start.
Delete Iter Calls	Delete iterator method calls since RC start.
Process Iter Calls	Process iterator method calls since RC start.
Errors	Errors detected during iterator processing calls since RC start.
EOF	Number of EOF conditions detected during iterator processing calls since RC start.
Invld ShmId	Number of Invalid SHM ID conditions detected during iterator processing calls since RC start.
Return Value	Iterator processing returns value since RC start.
Return Ptr	Iterator processing returns pointer since RC start.
Erase	Number of erased records during iterator processing since RC start.
UnLock	Number of unlocked hash buckets during iterator processing since RC start.
Next	Number of next-record operations conducted during iterator processing since RC start.

Session Mdb Garbage Collection Statistics describes garbage collection statistics.

Table 11. Session Mdb Garbage Collection Statistics
Statistic	Description
Timeout (ms)	Current garbage collection processing timeout in milliseconds.
Loops	Garbage collection processing loop counter since RC start.
Recs Erase	Records erased during garbage collection since RC start.

Session Mdb Main Info Object Statistics describes object statistics.

Table 12. Session Mdb Main Info Object Statistics
Statistic	Description
Total Recs	Total number of records.
Recs Avail	Total number of available records.
Recs Avail %	Percentage of all records that are available.

Route Cache Agent Statistics

Route Cache Agent Statistics describes route cache agent statistics.

Table 13. Route Cache Agent Statistics
Statistic	Description
rxBadSzMsg	RX Bad size message counter.
rxIncompatibleVer	RX Incompatible version counter.
rxInvalidMsg	RX Invalid message counter.
txDroppedPeerDisconnected	TX Dropped peer disconnected counter.
txErr	TX Error counter.
whoHasRx	RX Who has counter.
whoHasTx	TX Who has counter.
whoHasTxErr	TX Who has error counter.