- View System Logs
You can view several MATRIXX log files to gather information about runtime processing, including the mtx_debug.log, diameter_error_mdc.log, mdc_gateway_error_mdc.log, mtx_cmd.log, and storage_commands.log.
- Identifying Issues with System MDCs
In 511x and previous releases, system MDCs were included in the ${MTX_DATA_DIR}/mtx_config_base.xml file. System MDC definitions are now in the ${MTX_DATA_DIR}/mdc_config_system.xml. This file is not used by MATRIXX Digital Commerce itself; it is only provided to help in using these MDCs.
- Avoid Performance Issues Caused by Slow IO
In a production environment, avoid running IO-intensive operations, such as using gunzip to unzip large zip files; for example, to unzip zipped up MATRIXX log files. Running IO-intensive commands can cause IO spikes and degrade transaction server performance by causing transaction write failures.
- Logging Diameter and MDC Gateway Errors
Configure MATRIXX Engine to generate log files to hold Diameter Gateway and MDC Gateway errors.
- Analyzing the STANDBY Cluster INIT Phase
When in an HA state of INIT, a cluster is in the process of initializing its databases from a checkpoint or transaction log file (when the cluster will become the STANDBY cluster), or from replaying the in-memory databases (when the cluster will become the ACTIVE cluster). During this time, messages might be written to the mtx_debug.log file that indicate errors, and others that do not. These are listed below.
- Analyze Queue Latencies
Use the analyze_mtx_debug_log.py script to analyze the queue statistics in the mtx_debug.log file and aid in troubleshooting problems with transaction performance. Queue statistics are collected for the Diameter Gateway, MDC Gateway, Charging Server, and Transaction Server.
- Analyze a Segmentation Fault
Run the analyze_seg_fault.py script to gather information about a specified segfault line in a stack trace. The script prints the function name, source file name, and line number where the error occurred. of the segfault. This information can be sent to a MATRIXX representative for troubleshooting errors in the MATRIXX Engine code.
- Analyze a Stack Frame
Use the analyze_stack_frame.py script to analyze a stack frame from an error in the mtx_debug.log file or a specified text file and retrieve the filename and line number in which the error occurred. This information can be sent to a MATRIXX representative for troubleshooting errors in the MATRIXX Engine code.
- Analyze Transaction Replay Performance
Use the analyze_replay_performance.py script after a database initialization completes to analyze transaction replay throughput statistics, including the replay duration, any throttling, the number of transactions replayed, and the number of transactions replayed per second.
- Display Critical Log Messages
You can extract critical errors from a MATRIXX core file and save them to a specified output file.
- Filter Out Entries From the Debug Log
During system runtime, you can query the debug log for a particular string of text, including words, characters, and patterns of characters, and have all logical log entries containing that expression written to a specified output file. This is useful when the debug log is large and you need to isolate a specific area of functionality to understand and resolve problems.
- Fix Java-Related Out-of-Memory Problems
The default Java heap memory settings are appropriate for MATRIXX Engine Java tools in a normal MATRIXX Digital Commerce implementation. However, if you encounter Java-related "out of memory" error messages, you can follow the instructions in these topics to increase the Java heap size.
- Merge Messages from Multiple Debug Logs
You can merge the messages from multiple system logs into a single log file. The result is the combined log messages ordered according to the timestamp on the LM_ line.
- Resend Diameter Packets
Resend Diameter packets using a MATRIXX log file or a Wireshark trace file to verify that the packets are processed correctly or to recreate an issue encountered during processing.
- Dynamically Configure Runtime Debug/Trace Logging
You can change the default debug or trace level on any blade server during runtime, without changing the system configuration. This enables debugging and trace messages to be written to the system log. Messages can be written for the entire blade, a specific server in the blade, or a specific task running in any MATRIXX process. You specify the default logging level for a blade server at startup using the create_config.info file by using the question: What log level do you want to use? By default MATRIXX Engine logs information at the info log level.
- Turn on Trace Logging for a Subscriber
You can turn on trace logging for a subscriber during runtime by specifying a trace option such as a device IMSI or NAI (network access identifier). Turning on a trace sets the logging level for the specified gateway (Diameter Gateway, Camel Application Part (CAP) Gateway, or both), and the Charging Server dynamically during runtime. This allows these MATRIXX processes to log trace messages to the mtx_debug.log file.
- Correcting Too-Large Message Errors
MATRIXX Digital Commerce enforces a message size limit of 2097152 bytes to keep transactions from becoming too large to process effectively.
- Correcting Duplicate Checkpoint Errors
If MATRIXX Digital Commerce determines that the MATRIXX Engine is attempting a cold restart using a stale (inconsistent) checkpoint, it stops the process and alerts you with a duplicate checkpoint error message. In this case, the checkpointing server is out of out-of-sync with the latest snapshot, and you should investigate the issue and restart the checkpointing server before attempting to restart the engine.
