Home
MATRIXX Monitoring and Logging
Appendixes
Analytical and Informational Scripts
Use the scripts in this section to gather information about system performance, system topology, performance statistics, system configuration, and MATRIXX Engine versioning. All scripts run locally on a server as user mtx.
sort_mtx_debug_log.py
The sort_mtx_debug_log.py script sorts one or more specified log files by the timestamps found in the file(s) and prints them to standard output to aid in debugging. Most of the lines in a log file are in order but, due to multi-threading issues and the formatting needed by certain lines, these lines can be written in the wrong order.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
- About MATRIXX Monitoring and Logging
  MATRIXX Monitoring and Logging describes monitoring and logging for MATRIXX components in a MATRIXX environment.
- Metrics and Alerts
  The following information describes monitoring metrics and alerts information for MATRIXX. MATRIXX Support recommends using the Prometheus and Grafana third-party products to monitor a MATRIXX environment, along with aggregated logging.
- Logging
  The following describes logging information for MATRIXX.
- Tracing
  The following information describes tracing information for MATRIXX.
- Monitoring Components
  The following information describes monitoring different components for MATRIXX.
- Appendixes
  - System Log Configuration Files
    Use the system log scripts and configuration files to configure log rotation, gather system logs, set object tracing, and change the log level dynamically.
  - Analytical and Informational Scripts
    Use the scripts in this section to gather information about system performance, system topology, performance statistics, system configuration, and MATRIXX Engine versioning. All scripts run locally on a server as user mtx.
    - analyze_core_files.py
      The analyze_core_files.py script queries the specified input log for information related to critical errors and writes them to the specified output file.
    - analyze_diagstats.py
      The analyze_diagstats.py script analyzes each statistics file and reports any issues. For example, the analyze_diagstats.py script reports if a queue is full and is still incrementing, indicating a performance issue.
    - analyze_mtx_debug_log.py
      The analyze_mtx_debug_log.py script analyzes the lines in MATRIXX debug log files.
    - performance_analysis_event_server.py
      This script analyzes mtx_event_streamer-nn.log files for Event Stream Server performance, estimating processing latencies and first delivery times for events. This script produces CSV files and corresponding graphs (PNG files) from the CSV data to aid in analysis.
    - analyze_process.py
      The analyze_process.py script collects statistics for a running MATRIXX process and writes the information to two log files.
    - analyze_replay_performance.py
      After a database initialization completes, run the analyze_replay_performance.py script on the processing server that serviced the replay request to analyze transaction replay throughput statistics, including the checkpoint creation time, any throttling, the number of transactions replayed, and the duration to replay them. Look for database initialization requester or start sending in-memory database replay to peer cluster log messages to find the processing server on which to run the script. The script uses the mtx_debug.log file to analyze the replay performance because the server where it is run drove the replay, both locally and remotely.
    - analyze_seg_fault.py
      The analyze_seg_fault.py takes a line in a stack frame and prints the function name, source filename, and line number where the error occurred. This information can be sent to a MATRIXX Support representative to help troubleshoot errors.
    - analyze_stack_frame.py
      The analyze_stack_frame.py script takes a line in a stack frame in the mtx_debug.log or in a specified file and prints the function name, the source filename, and the line number of this stack frame. This information can be sent to a MATRIXX representative for troubleshooting errors.
    - capture_diagstats_tofile.py
      The capture_diagstats_tofile.py script runs automatically on MATRIXX Engine, collecting SNMP processing statistics for diagnostic use. By default, it runs every five minutes and at each 24-hour increment, it rotates the compressed statistics file and starts collecting statistics in a new file.
    - check_engine_start_prereqs.py
      Run the check_engine_start_prereqs.py script before starting MATRIXX Engine or a cluster to check the start-up prerequisites for the local transaction log directories.
    - check_system_configuration.py
      The check_system_configuration.py script checks and can correct system configuration settings for MATRIXX components. It runs on individual components as they are installed and on MATRIXX Engine when you start it by itself. You can also run this script as needed on components of an installed system to check or correct system configuration. The behavior of this script is limited to the permissions of the user running it. Running it as a user with non-sudo permissions just alerts you to configuration issues on file systems and files that the user as permission to read. Running this script with sudo permissions and the --autocorrect=1 option gives it access to more settings and directs it to correct any configuration issues it finds.
    - cluster_mgr_cli.py
      The cluster_mgr_cli.py script provides a simple command line client that can be run on an off-engine server such as a Network Operation Center (NOC) to manage certain cluster operations. This script can retrieve cluster and peer cluster HA states, low-level information about cluster status, the Cluster Manager leader, and the cluster schema version. This script can also shut down a target cluster.
    - gather_logs.py
      The gather_logs.py script collects system logs from the ${MTX_LOG_DIR} directory, core files, and custom configuration files that are used at runtime and adds them to a ZIP file so they can be offloaded to a non-engine sever. The gather_logs.py script also analyzes the data in the core file and writes the results to a file with the same name but with a .log extension.
    - normalize_mtx_debug_log.py
      The normalize_mtx_debug_log.py script removes fields and lines from log files that have variable information (such as memory addresses, time stamps, and PIDs) and writes the modified lines to an output file.
    - plot_diagstats.py
      The plot_diagstats.py script reads the statistics files from the current directory and plots the collected data by specified Management Information Base (MIB) files and time range. This script plots a maximum of six MIB and object IDs (OIDs) at a time.
    - print_blade_stats.py
      The print_blade_stats.py script displays statistics for MATRIXX Engine, in-memory databases, cluster and server states, system memory, notifications, Task Manager, Diameter, MDC Gateway, CCF statistics, checkpoint statistics (such as the server ID, last or in-progress checkpoint state, checkpoint start and end times, type of checkpoint (ad hoc, fast restart), related _glossary/global_transaction_counter.html, and checkpoint name and path), system monitor, and others. For a description of all SNMP statistics, see the discussion about all MATRIXX SNMP statistics in MATRIXX Monitoring and Logging.
    - print_data_container_file.py
      The print_data_container_file.py script prints the contents of a MATRIXX data container (MDC) file in XML format. The file can be a transaction log file, _glossary/mef.html, checkpoint file, or any MDC.
    - print_data_container_pickled_file.py
      The print_data_container_pickled_file.py script is a debugging script that prints the information stored in the ${MTX_CONF_DIR}/mdc_config_custom.pickled file (which holds the custom MATRIXX Data Containers (MDCs) or the ${MTX_DATA_DIR}/data_container/mdc_config_system.pickled file (which holds the system MDCs).
    - print_diagstats.py
      The print_diagstats.py script reads each statistics file and prints out the result.
    - print_gw_connections.py
      The print_gw_connections.py script reports current connections maintained by Diameter Gateway or MDC Gateway.
    - print_in_use_mtxbufs
      The print_in_use_mtxbufs script provides information about the in-use memory buffers on a server. It mainly provides details about what is inside the MtxBuf buffers that are in-use.
    - print_mtx_version.py
      The print_mtx_version.py script prints the current version information, or prints a MATRIXX JAR or WAR file, if given.
    - print_topology.py
      Use this script to display information about software topology, including sites, domains, load balancers, shared storage, MATRIXX engine IDs, cluster IDs, server IDs, and IP addresses.
    - print_snmp_stats.py
      The print_snmp_stats.py script prints statistics for all MATRIXX Engine sub-domains configured for a domain, the Route Cache Controller, Route Cache Agent, Traffic Routing Agent HA pairs, Network Enabler, and any related MATRIXX Peer Manager instances, depending on the specified options. Run this script as the tra user on the server hosting any of these MATRIXX components.
    - print_sysmon_info.py
      The print_sysmon_info.py script retrieves information for monitored objects.
    - print_tra_connections.py
      The print_tra_connections.py script reports active Traffic Routing Agent (TRA) connections data through the management interface server for all virtual servers. Enable connections reporting in the tra_config.xml file.
    - set_subscriber_trace.py
      The set_subscriber_trace.py script turns on tracing, which changes the log level for the specified gateway (Diameter Gateway, CAMEL Application Part (CAP), or both), and the Charging Server dynamically during runtime. Turning on tracing allows these MATRIXX processes to log trace messages to the mtx_debug.log file. The script applies to all servers in the specified engine.
    - set_trace.py
      The set_trace.py script changes the system log level dynamically for a single MATRIXX Engine server during runtime for the current session. Specific messages can be logged for one or more MATRIXX services, including more specific tasks that form a service. The script updates the values set in the ${MTX_CUSTOM_DIR}/mtx_log_config_update.cfg file but does not change the base system configuration parameter in the ${MTX_CONF_DIR}/mtx_config.xml file. The script only makes logging changes on the server on which the script runs.
    - sort_mtx_debug_log.py
      The sort_mtx_debug_log.py script sorts one or more specified log files by the timestamps found in the file(s) and prints them to standard output to aid in debugging. Most of the lines in a log file are in order but, due to multi-threading issues and the formatting needed by certain lines, these lines can be written in the wrong order.
    - split_mtx_debug_log.py
      The split_mtx_debug_log.py script reads the specified MATRIXX debug log file, retrieves all logical log entries that match the specified input parameter, and writes them to the specified output file.
    - start_collecting_stats.py
      Use this script to collect performance statistics. The current statistics, before the script runs, are written to the ${MTX_LOG_DIR}/netstats_before.log file. After running this script, run the stop_collecting_stats.py script to stop the statistics collection and write the data to the ${MTX_LOG_DIR}/netstats_after.log file. The differences between the before and after data is written to the ${MTX_LOG_DIR}/netstats_diff.log file.
    - stop_collecting_stats.py
      Use this script to stop the statistics collection operation triggered by the start_collecting_stats.py script and close each log file being written. This script also writes the statistics to the ${MTX_LOG_DIR}/netstats_after.log file and writes the differences between before and after the script is run to the ${MTX_LOG_DIR}/netstats_diff.log file.
    - trace_object.py
      Sets a trace counter on the specified object and returns the object.
    - validateCheckpoint.jar
      The validateCheckpoint.jar program analyzes a MATRIXX checkpoint, detects any errors in the database, and produces a validation report listing internal database statistics and any errors found. This information can be sent to a MATRIXX representative to help troubleshoot database errors.
  - Configuration, Demo, and Test Scripts
    Use these scripts to configure MATRIXX Engine, create demos, and test system functionality.
  - Dynamic Logging Scripts
    Use the system log scripts to change the log level dynamically.
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

sort_mtx_debug_log.py

The sort_mtx_debug_log.py script sorts one or more specified log files by the timestamps found in the file(s) and prints them to standard output to aid in debugging. Most of the lines in a log file are in order but, due to multi-threading issues and the formatting needed by certain lines, these lines can be written in the wrong order.

Syntax

sort_mtx_debug_log.py [-f filename | -h | -d | --debug_performance]

Options

-f filename, --file=filename: Runs the analysis on the specified system log file or files. Log files are written to the directory specified with the ${MTX_LOG_DIR} environment variable. When you have multiple input files, each input file is processed entirely before the next file is processed and only one output file is generated. Therefore, the output is sorted per input file, but not necessarily sorted based upon the entire output file. It depends on the order the files are processed and the contents of the files.
-h, --help: Help about the script options.
-d, --debug: Debug flag. The default is off. If specified, extra messages are printed. It can be specified multiple times, which increases the types of extra messages printed.
--debug_performance: Do performance timings on this script.

For more information about the MATRIXX environment variables, see the discussion about container directories and environment variables in MATRIXX Installation and Upgrade.

Return More Debug Log Information

sort_mtx_debug_log.py -f ${MTX_LOG_DIR}/mtx_debug.log -d

Sort Messages in the transaction_server.stderr.log File

sort_mtx_debug_log.py -f ${MTX_LOG_DIR}/transaction_server.stderr.log