Home
MATRIXX Monitoring and Logging
Monitoring Components
The following information describes monitoring different components for MATRIXX.
Monitoring MATRIXX Call Control Framework
The topics in this section have information about monitoring Call Control Framework (CCF) components, including Network Enabler and CAMEL Gateway.
Call Control Framework SNMP Statistics
SNMP statistics are defined in the MATRIXX Engine MIB files. Call Control Framework statistics are returned by the print_snmp_stats.py script run on both the processing servers and on the Network Enabler servers. They include statistics on signaling network links, TCAP messages, CAMEL Gateway sessions, voice charging and SMS charging, MAP call outs, USSD notifications, USSD incoming service, and USSD call back services.
Display all Call Control Framework Statistics
Display statistics for components of Call Control Framework (CCF) to check that Transaction Capabilities Application Part (TCAP) protocol messages are being sent and received, and to check that calls, announcements, and SMS messages are being processed. See the discussion about MATRIXX SNMP Statistics in MATRIXX Monitoring and Logging for a complete list of the CCF statistics.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
- About MATRIXX Monitoring and Logging
  MATRIXX Monitoring and Logging describes monitoring and logging for MATRIXX components in a MATRIXX environment.
- Metrics and Alerts
  The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
- Logging
  The following describes logging information for MATRIXX.
- Tracing
  The following information describes tracing information for MATRIXX.
- Monitoring Components
  The following information describes monitoring different components for MATRIXX.
  - Monitoring MATRIXX Business API Gateway
    Business API Gateway SNMP statistics are defined in the matrixx_web_mib.txt file and you display them using the snmp-net-utils snmpwalk command.
  - Monitoring MATRIXX Call Control Framework
    The topics in this section have information about monitoring Call Control Framework (CCF) components, including Network Enabler and CAMEL Gateway.
    - Call Control Framework SNMP Statistics
      SNMP statistics are defined in the MATRIXX Engine MIB files. Call Control Framework statistics are returned by the print_snmp_stats.py script run on both the processing servers and on the Network Enabler servers. They include statistics on signaling network links, TCAP messages, CAMEL Gateway sessions, voice charging and SMS charging, MAP call outs, USSD notifications, USSD incoming service, and USSD call back services.
      - Display all Call Control Framework Statistics
        Display statistics for components of Call Control Framework (CCF) to check that Transaction Capabilities Application Part (TCAP) protocol messages are being sent and received, and to check that calls, announcements, and SMS messages are being processed. See the discussion about MATRIXX SNMP Statistics in MATRIXX Monitoring and Logging for a complete list of the CCF statistics.
      - Display Network Enabler Statistics
        Network Enabler statistics include the number of connection bridges and the total number of upstream and downstream bytes sent and received.
    - Sample Debug Logs for Call Control Framework
      This section shows example debug logs for Call Control Framework (CCF).
    - Call Control Framework Monitoring Messages
      ccf/r_ccf_ne_messages.html#reference_inn_hsg_bdc__table_dqh_gvc_cnb provides a list of messages related to Call Control Framework (CCF) and Network Enablers (NEs). Use this information to configure the appropriate alerts based on their severity and potential impact.
  - Monitoring Diameter Events
    Diameter Event Logger listens to the Diameter Gateway through a TCP connection and logs diameter application messages to a JSON file.
  - Monitoring Event Streaming
    Event streaming errors are logged in the Event Streaming Framework mtx_event_streamer.log file. Event Streaming Framework uses Log4j 2.x to log error, debugging, and informational messages. You can also monitor event streaming performance using the performance_analysis_event_server.py script. For information about the script, see the discussion about performance_analysis_event_server.py.
  - Monitoring MATRIXX Route Cache
    Monitor the Route Cache by reading watermark log messages for each Route Cache table and Route Cache SNMP statistics. The SNMP statistics are defined in the matrixx_rc_mib.txt MIB file and returned by the print_snmp_stats.py script. They include statistics on subscriber and session data, statistics on insert, fetch, erase, and iterator operations, general statistics on subscriber records, and garbage collection.
  - Monitoring Payment Service Logging
    Payment service logs messages to the payment-service.log.
  - Monitoring MATRIXX SNMP Statistics
    By default, the SNMP Agent running on a MATRIXX Engine monitors the status and health of all the servers and clusters it has. It collects statistics to aid in identifying potential processing issues.
  - Monitoring MATRIXX Traffic Routing Agent
    Traffic Routing Agent SNMP statistics are defined in the matrixx_tra_mib.txt MIB file and are returned by the print_snmp_stats.py script when run on any of the Traffic Routing Agent servers. They include categories such as TCP traffic, UDP traffic, Diameter Gateway traffic, and MDC Gateway traffic.
  - Monitoring MATRIXX Web Apps
    Use the MATRIXX system logs and SNMP statistics and notifications to monitor the MATRIXX web-facing applications, such as RS Gateway, Gateway Proxy, and My MATRIXX.
  - Monitoring MATRIXX SBA Gateway
    The MATRIXX SBA Gateway SNMP statistics are defined in the matrixx_web_mib.txt file, using the snmp-net-utils snmpwalk command to display them.
  - Monitoring 5G Event Streaming
    5G event streaming can be monitored with standard Kafka and custom metrics.
  - Monitoring Kafka CDR Consumer
    Use aggregated logging and the Prometheus and Grafana third-party products to monitor Kafka CDR Consumer.
- Appendixes
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Display all Call Control Framework Statistics

Display statistics for components of Call Control Framework (CCF) to check that Transaction Capabilities Application Part (TCAP) protocol messages are being sent and received, and to check that calls, announcements, and SMS messages are being processed. See the discussion about MATRIXX SNMP Statistics in MATRIXX Monitoring and Logging for a complete list of the CCF statistics.

About this task

For a complete list of the SNMP statistics, see the discussion about all MATRIXX SNMP statistics. For details about the print_blade_stats.py script, see the discussion about print_blade_stats.py.

The following CCF statistics are tracked:

Signaling network statistics for messages received and sent per signaling link, and the current state of external links between Network Enabler and the network.
TCAP statistics for sending and receiving TCAP protocol messages.
CAMEL (Customised Applications for Mobile network Enhanced Logic) Gateway session statistics.
Voice Charging statistics for calls, announcements, and VXML scripts.
SMS Charging statistics for CAP3 (CAMEL Application Part 3) short messages sent and received.
TSAN (Temporary Subscriber Access Number) statistics for the CAP1 re-origination service for the number of TSAN requests, timeouts, successful releases and bad messages.
MAP (Mobile Application Part) protocol statistics for MAP-ATI (anytime interrogation) and MAP-SRI (send routing information) callout requests and responses.
USSD (Unstructured Supplementary Service Data) notification statistics for MAP USSD Notify requests and responses.
USSD incoming service statistics for USSD notifications requested and sent.
USSD call back statistics for successful and failed call back requests.
Call back call start statistics for successful and failed call start attempts.

Perform these steps on a processing server to display statistics for CCF.

Procedure

To view statistics on signaling messages sent and received per signaling link and to view the current state of external links between Network Enabler and the network, enter the following command.

print_blade_stats.py -F

You see statistics similar to the following:

$ print_blade_stats.py -F 

Signalling Network Stats
------------------------

                     Received  Received     
                     Rate      Delivery   Payload    UDT                  UDTS                                   
Link Name   State    Limit     Error      Data Sent  Received  UDT Sent   Received  UDTS Sent 
=============================================================================================
link_ne1   ASP_UP           0         0       10000     10000     10000      10000          0                
link_ne2   ASP_UP           0         0       10000     10000     10000      10000          0     

XUDT                 XUDTS                  LUDT                 LUDTS     LUDTS
Received  XUDT Sent  Received   XUDTS Sent  Received  LUDT Sent  Received   Sent
      
=================================================================================
       0          0         0            0         0          0         0      0
       0          0         0

To view statistics on TCAP messages sent and received, enter the following command.

print_blade_stats.py -J

You see statistics similar to the following:

TCAP Stats
------------------
                                                        
Messages Received                  | Messages Sent             
Uni Begin  End Continue Abort UDTS | Uni  Begin  End Continue Abort
=================================================================== 
  0   143  134      157     4    1     0      0    4      161     1 


Received Sent              ASN.1    ASN.1
Unknown  Unknown Rejected  Decoding Encoding
Txn Id   Txn Id  Overload  Error    Error

=============================================
     0        0         0       0        0

To view CAMEL Gateway statistics, enter the following command.

print_blade_stats.py -L

You see statistics similar to the following:

CAMEL Gateway Stats
---------------------------

    Charging Sessions   
Started   Ended   Error 
==================================
    142     142       1

To view Voice Charging statistics, enter the following command.

print_blade_stats.py -V

You see statistics similar to the following:

Voice Charging Stats
--------------------
                      |                       |                      |                      | 
 InitialDPs Received  |      ACs  |   ACRs    |   Time               |    Calls Rejected    |
     Valid    Invalid |      Sent |  Received |   Granted       Free | No Credit      Other |
=============================================================================================
         3          0           3           3           2          0           0          1           

    |   Charging Failure   |                      |        Timed Out waiting         |
    |      Treatment       |  Calls Disconnected  |         For MSC Response         |
    | InitialDP        ACR | No Credit      Other |      ACR     ERBCSM ActivityTest |
    =================================================================================
             0          0           0          0          0          0            0

Voice Charging Call Outcome Stats
---------------------------------
                           Route                                  |
                          Select            Caller     Called     |
 Abandoned       Busy    Failure  No Answer Disconnect Disconnect |
===================================================================
         0          0          0          0          2          0


Voice Charging Announcement and VXML Script Stats
-------------------------------------------------
                      |                      |        Failed         |                                   |
    Announcements     |     VXML Scripts     | Connection       IVR  |     Timed Out         |           |
 Attempted  Completed | Attempted  Completed |      Error      Error | Connecting    Playing | Abandoned |
==========================================================================================================
         0          0           0          0            0          0            0          0           0

To view CAP3 SMS Charging statistics, enter the following command.

print_blade_stats.py -M

You see statistics similar to the following:

CAP3 SMS Charging Stats
-----------------------

 InitialDPSMSs Received | Immediate |    Unit     |    SMS Rejected   |
     Valid      Invalid |  Charge   | Reservation | No Credit   Other |
=======================================================================
         2            0           1             0         0          1


      SMS Submit      |   ERSMS   |  Network |
 Succeeded     Failed |   Timeout |   Error  |
===============================================
        0          0           0          0

To view TSAN statistics for the CAP1 reorigination service, enter the following command.

print_blade_stats.py -O

You see statistics similar to the following:

TSAN Stats 
----------------------------------------------------- 
Requests Timeout Successful Rel Bad Message 
===================================================== 
       1       1              0           0

To view MAP-ATI and MAP-SRI statistics, enter the following command.

print_blade_stats.py -A

You see statistics similar to the following:

MAP Call Out AnyTimeInterrogation Stats
---------------------------------------
  Requests    Success    Timeout      Error      Abort  Routing Error  System Failure
=====================================================================================
         5          5          0          0          0              0               0

MAP Call Out SendRoutingInfoStats
---------------------------------
  Requests    Success    Timeout      Error      Abort  Routing Error  System Failure
=====================================================================================
         0          0          0          0          0              0               0

To view USSD Notifications statistics, enter the following command.

print_blade_stats.py -U

You see statistics similar to the following:

USSD Call Out Ussd Query Stats
---------------------------------------
  Requests    Success    Timeout      Error      Abort  Routing Error  System Failure
=====================================================================================
         1          0          0          1          0              0               0

To view USSD incoming service statistics, enter the following command.

print_blade_stats.py -X

You see statistics similar to the following:

USSD Incoming Stats
---------------------------------------
  Requests    Sent Mtx Notification      Error Mtx Notification
=====================================================================================
         1              1                    0

Enter the following command to view call back call start statistics for the voice charging service.

print_blade_stats.py -H

You see statistics similar to the following:

Call Start Stats part 1
-----------------------------------------------------------------------------------------------
       Requests        Attempts   Start success     Sys failure         Bad Msg   Route failure
===============================================================================================
              1               1               1               0               0               0


Call Start Stats part 2
-----------------------------------------------------------------------------------------------
      No Credit      Unknow Sub  Unknown Reason        Time out  Calling No Ans    Calling Busy
===============================================================================================
              0               0               0               0               0               0


Call Start Stats part 3
-----------------------------------------------------------------------------------------------
    Calling RSF    Calling Disc   Called No Ans     Called Busy      Called RSF     Called Disc
===============================================================================================
              0               1               0               0               0               0

Enter the following command to view the USSD call back statistics.

print_blade_stats.py -I

You see statistics similar to the following:

USSD Call Back Ussd Stats
-------------------------------------------------------------------
       Requests-Notif   Bad Msg   Start success   Start failure    
===================================================================
              1               0               1               0