Home
MATRIXX Monitoring and Logging
Logging
The following describes logging information for MATRIXX.
Using Logs to Troubleshoot Problems
This section contains information to help identify and resolve system problems.
Message Identifiers in Log Files
Log messages related to many internal and external messages are tagged with a unique RelatedMsgId identifier by default. The RelatedMsgId allows you to correlate a message throughout MATRIXX.
Generate a List of RelatedMsgIds
Perform this task to generate a list of logged messages with a unique RelatedMsgId from a log file.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
- About MATRIXX Monitoring and Logging
  MATRIXX Monitoring and Logging describes monitoring and logging for MATRIXX components in a MATRIXX environment.
- Metrics and Alerts
  The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
- Logging
  The following describes logging information for MATRIXX.
  - Configuring MATRIXX Engine Logging
    The topics in this section provide information about maintaining, monitoring, and resolving problems with a MATRIXX Engine installation.
  - Network Enabler Logging in Kubernetes
    Network Enabler (NE) logs in a MATRIXX installation for Docker and Kubernetes must be written to a persistent volume so that they are retained if a NE container is re-created.
  - Changing System Logging Behavior
    Follow the procedures in this section to change the default rsyslog behavior and the log level for a MATRIXX business application or component.
  - Logging and Alerts
    Cloud native MATRIXX installations can replace or remove containers in the process of scaling or failure recovery. This presents challenges such as insuring log data persists after a container has been removed or replaced, and requiring to be able to view and search logs without having to log in to many individual containers.
  - Using Logs to Troubleshoot Problems
    This section contains information to help identify and resolve system problems.
    - View System Logs
      You can view several MATRIXX log files to gather information about runtime processing, including mtx_debug.log, diameter_error_mdc.log, mdc_gateway_error_mdc.log, mtx_cmd.log, and storage_commands.log.
    - Logging Diameter and MDC Gateway Errors
      Configure MATRIXX Engine to generate log files to hold Diameter Gateway and MDC Gateway errors.
    - Message Identifiers in Log Files
      Log messages related to many internal and external messages are tagged with a unique RelatedMsgId identifier by default. The RelatedMsgId allows you to correlate a message throughout MATRIXX.
      - Generate a List of RelatedMsgIds
        Perform this task to generate a list of logged messages with a unique RelatedMsgId from a log file.
    - Display Critical Log Messages
      You can extract critical errors from a MATRIXX core file and save them to a specified output file.
    - Filter Out Entries From the Debug Log
      During system runtime, you can query the debug log for a particular string of text, including words, characters, and patterns of characters, and have all logical log entries containing that expression written to a specified output file. This is useful when the debug log is large and you must isolate a specific area of functionality to understand and resolve issues.
    - Merge Messages from Multiple Debug Logs
      You can merge the messages from multiple system logs into one log file. The result is the combined log messages ordered according to the time stamp on the LM_ line.
    - Dynamically Configure Runtime Debug/Trace Logging
      You can change the default debug or trace level on any server during runtime without changing the system configuration. This process enables debugging and trace messages to be written to the system log. Messages can be written for the entire server or a specific task running in any MATRIXX process. You specify the default logging level for a server at start-up using the create_config.info file by using the question: What log level do you want to use?. By default MATRIXX Engine logs information at the informational log level.
    - Turn on Trace Logging for a Subscriber
      You can turn on trace logging for a subscriber during runtime by specifying a trace option such as a device IMSI or NAI (network access identifier). Turning on a trace sets the logging level for the specified gateway (Diameter Gateway, CAMEL Application Part (CAP), or both) and the Charging Server dynamically during runtime. This allows these MATRIXX processes to log trace messages to the mtx_debug.log file.
  - Java-Based Container Logging
    Log output from applications running in containers must be written to persistent storage outside of the container to be preserved when the container is recreated after a failure or at application start.
  - Topology Operator Logging
    Topology Operator and related components can be configured to log to STDOUT or a persistent volume (PV), similar to other MATRIXX components.
- Tracing
  The following information describes tracing information for MATRIXX.
- Monitoring Components
  The following information describes monitoring different components for MATRIXX.
- Appendixes
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Generate a List of RelatedMsgIds

Perform this task to generate a list of logged messages with a unique RelatedMsgId from a log file.

Procedure

In a terminal, enter the following command to go to the system log directory:
cd /var/mtx
Enter the following command, replacing log_file with the name of the log file to parse:
cat log_file | awk '{match($0,"] ([^[:space:]]+) ",a)} {print a[1]}' | sort | uniq