Home
MATRIXX Monitoring and Logging
Logging
The following describes logging information for MATRIXX.
Using Logs to Troubleshoot Problems
This section contains information to help identify and resolve system problems.
Logging Diameter and MDC Gateway Errors
Configure MATRIXX Engine to generate log files to hold Diameter Gateway and MDC Gateway errors.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
- About MATRIXX Monitoring and Logging
  MATRIXX Monitoring and Logging describes monitoring and logging for MATRIXX components in a MATRIXX environment.
- Metrics and Alerts
  The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
- Logging
  The following describes logging information for MATRIXX.
  - Configuring MATRIXX Engine Logging
    The topics in this section provide information about maintaining, monitoring, and resolving problems with a MATRIXX Engine installation.
  - Network Enabler Logging in Kubernetes
    Network Enabler (NE) logs in a MATRIXX installation for Docker and Kubernetes must be written to a persistent volume so that they are retained if a NE container is re-created.
  - Changing System Logging Behavior
    Follow the procedures in this section to change the default rsyslog behavior and the log level for a MATRIXX business application or component.
  - Logging and Alerts
    Cloud native MATRIXX installations can replace or remove containers in the process of scaling or failure recovery. This presents challenges such as insuring log data persists after a container has been removed or replaced, and requiring to be able to view and search logs without having to log in to many individual containers.
  - Using Logs to Troubleshoot Problems
    This section contains information to help identify and resolve system problems.
    - View System Logs
      You can view several MATRIXX log files to gather information about runtime processing, including mtx_debug.log, diameter_error_mdc.log, mdc_gateway_error_mdc.log, mtx_cmd.log, and storage_commands.log.
    - Logging Diameter and MDC Gateway Errors
      Configure MATRIXX Engine to generate log files to hold Diameter Gateway and MDC Gateway errors.
    - Message Identifiers in Log Files
      Log messages related to many internal and external messages are tagged with a unique RelatedMsgId identifier by default. The RelatedMsgId allows you to correlate a message throughout MATRIXX.
    - Display Critical Log Messages
      You can extract critical errors from a MATRIXX core file and save them to a specified output file.
    - Filter Out Entries From the Debug Log
      During system runtime, you can query the debug log for a particular string of text, including words, characters, and patterns of characters, and have all logical log entries containing that expression written to a specified output file. This is useful when the debug log is large and you must isolate a specific area of functionality to understand and resolve issues.
    - Merge Messages from Multiple Debug Logs
      You can merge the messages from multiple system logs into one log file. The result is the combined log messages ordered according to the time stamp on the LM_ line.
    - Dynamically Configure Runtime Debug/Trace Logging
      You can change the default debug or trace level on any server during runtime without changing the system configuration. This process enables debugging and trace messages to be written to the system log. Messages can be written for the entire server or a specific task running in any MATRIXX process. You specify the default logging level for a server at start-up using the create_config.info file by using the question: What log level do you want to use?. By default MATRIXX Engine logs information at the informational log level.
    - Turn on Trace Logging for a Subscriber
      You can turn on trace logging for a subscriber during runtime by specifying a trace option such as a device IMSI or NAI (network access identifier). Turning on a trace sets the logging level for the specified gateway (Diameter Gateway, CAMEL Application Part (CAP), or both) and the Charging Server dynamically during runtime. This allows these MATRIXX processes to log trace messages to the mtx_debug.log file.
  - Java-Based Container Logging
    Log output from applications running in containers must be written to persistent storage outside of the container to be preserved when the container is recreated after a failure or at application start.
  - Topology Operator Logging
    Topology Operator and related components can be configured to log to STDOUT or a persistent volume (PV), similar to other MATRIXX components.
- Tracing
  The following information describes tracing information for MATRIXX.
- Monitoring Components
  The following information describes monitoring different components for MATRIXX.
- Appendixes
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Logging Diameter and MDC Gateway Errors

Configure MATRIXX Engine to generate log files to hold Diameter Gateway and MDC Gateway errors.

The Diameter Gateway log has Diameter message MATRIXX Data Containers (MDCs) that report a non-successful result (a Diameter result value other than 2001). Logging Diameter MDCs is useful in troubleshooting when a high-level service operation is successful but a child service context is not. Diameter error MDC logs are in compact MDC (CMDC) format and are written to the ${MTX_ERROR_MDC_LOG_DIR}. The MDC Gateway log has message MDCs sent from the MDC Gateway to a network application that report a non-successful result (a nonzero Result field value). The MDCs are typically SubMan API messages but can also be any MDC based on the MtxMsg MDC. MDC Gateway error MDC logs are in CMDC format and are written to the ${MTX_ERROR_MDC_LOG_DIR}.

Setting up Diameter Gateway and MDC Gateway logging is part of the overall system configuration. It involves enabling the logging, specifying an error log location, and specifying how closing of the logs is triggered.

For information about configuring Diameter and MDC Gateway logs, see the discussion about Diameter configuration parameters and MDC Gateway configuration in MATRIXX Installation and Upgrade.

For more information about the MATRIXX environment variables, see the discussion about installation directories, navigation shortcuts, and environment variables in MATRIXX Installation and Upgrade.

You cannot configure the Diameter and MDC Gateway log files to be rotated, but you can configure a crontab to remove them. For example:

# delete MDC GW error files older than 7 days 30 
1 * * * /bin/find /var/log/mtx/ -name 'mdc_gateway_error*' -type f -mtime +7 | xargs rm -rf 

# delete Diameter error files older than 7 days 30 
1 * * * /bin/find /var/log/mtx/ -name 'diameter_error_mdc*' -type f -mtime +7 | xargs rm -rf