MATRIXX Log Files

Depending on the MATRIXX Engine configuration, you can view MATRIXX log files to gather information about runtime processing.

Unless you specify another location, or another default location is specified in the log files listed in MATRIXX Log Files, all files are in /var/log/mtx/ by default.

Most log files include variable information, such as memory addresses, time stamps, PIDs, and so forth. Use the normalize_mtx_debug_log.py to remove this variable information to make it easier to share log files. For more information, see the discussion about the normalize_mtx_debug_log.py script in the MATRIXX Monitoring and Logging appendices.

For more information about the MATRIXX environment variables, see the discussion about container directories and environment variables in MATRIXX Installation and Upgrade.

Table 1. MATRIXX Log Files
File Description Troubleshooting
mtx_debug.log Contains information about operations performed during runtime, operational issues, and configuration settings. For processing pods, monitor:
  • LM_CRIT — Alarm raised on all occurrences.
  • LM_ERROR, LM_WARN — Alarm raised only on occurrences not included in the exclusion list.
For example: 
LM_ERROR 3528|3578 2014-05-07 
03:25:12.065864 
[transaction_server_1:1:1:1] 
TXN1-TransactionManager:
transaction_manager_task::
TransactionManagerTask::
handleInitDatabaseMsg: Could 
not initialize databases: 
result=14, message =
DataContainer: size=717, 
bufSize=717, bufMaxSize=32768,
this=0x7f1a1f097580
diameter_​error_mdc.log Contains Diameter message MDCs that report a non-successful result (a Diameter result value other than 2001). Logging Diameter MDCs is useful for troubleshooting when a high-level service operation is successful, but a child service context is not. A new file is created every minute if there are new error messages. 
# START - 06/04/2014 16:14:30 -
diameter_error_mdc_1_1_1_2014_06_04_16_
14_30_21306_1.log
<1,10,0,447,4510,2>[0:1:5:7,(2:12),
0:1:5:8,{<1,2,1,321,4510,2>[0,(2:12)],
<1,2,1,321,4510,2>[1,(2:12)]}
,12,12,,0:1:5:9]<1,20,0,94,4510,2>
[0,4,272,(31:MATRIXX-QA-TESTING.;1;
100957376),,,,318232796455143267,1,
(9:admard01l),(7:matrixx),
(13:matrixxsw.com),
(29:CsvFrameWork-Gy.matrixxsw.com),,4,
,,,2]<1,5,0,411,4510,2>[2,0]
<1,19,0,93,4510,2>
[2014-06-04T16:31:46.693302-07:00,
268500992,2,{1401924706693302,
1401924706693653,1401924706694648,,,
1401924706694649,1401924706694765,
1401924706695140,,,,,,,,,,,,,,,,,,
1401924706695942,,,,
1401924706695958,1401924706696366,
1401924706696654},1,,,29,0,
(83:PreRatingTask::handleInput: 
Did not compute service type object ID
 for this message),7.0:4.20,,,,1001,
2001,,,5005]
mdc_gateway​_error_mdc.log Contains message MDCs sent from the MDC Gateway to a network application that report a non-successful result (a nonzero Result field value). The MDCs are typically SubMan API messages, but they can also be any MDC based on the MtxMsg MDC. A new file is created every minute if there are new error messages. 
# START - 06/16/2014 10:27:48 -
 mdc_gateway_error_mdc.log
<1,4,0,238,4510,2>[,<1,0,1,378,4510,2>
[]<1,3,0,235,4510,2>[]]<1,20,0,94,4510,
2>[]<1,5,0,411,4510,2>[]<1,19,0,93,
4510,2>[2014-06-16T17:27:48.488333Z,17,
,{,,,1402939668488333,1402939668488357,
,,,,,,,,,,,,,,,,,,,,,,1402939668488706,
1402939668488734,,,},,,,42,0,
(132:FromNetworkTask::handleInput: 
Error: Received from addr=
127.0.0.1:4060, socket fd=17. 
This system is not active, returning an
 error.),0.0:0.0]
mtx.log The rsyslog utility logs all messages for a MATRIXX process to mtx.log.

Use the mtx_log_config_update.cfg configuration file to change the system log level dynamically during runtime to turn on debug and trace messages for MATRIXX processes and specific server tasks.
gateway_​proxy.log Gateway Proxy messages are based on the sections of code that produce the message and the severity of the message:
  • debug
  • warn
  • error

Log settings are configured in /opt/mtx/conf/log4j2-mtx_gateway_proxy.xml.

To customize Gateway Proxy logging, copy /opt/mtx/data/log4j2-mtx_gateway_proxy.xml to /opt/mtx/conf/log4j2-mtx_gateway_proxy.xml and modify the file in the new location. Revert changes by copying from /opt/mtx/data again if necessary.

Monitor the number of errors for a time interval.
mymatrixx.log Log settings are configured in /opt/mtx/conf/log4j2-matrixx.xml.

To customize My MATRIXX logging, copy /opt/mtx/data/log4j2-matrixx.xml to /opt/mtx/conf/log4j2-matrixx.xml and modify the file in the new location. Revert changes by copying from /opt/mtx/data again when necessary.
matrixxbct.log Log settings are configured in /opt/mtx/conf/log4j2-matrixxbct.xml.

To customize MATRIXX Backoffice Customer Tool logging, copy /opt/mtx/data/log4j2-matrixxbct.xml to /opt/mtx/conf/log4j2-matrixxbct.xml and modify the file in the new location. Revert changes by copying from /opt/mtx/data again if necessary.

The location of the log file is specified by MTX_LOG_DIR, which is normally set as -DMTX_LOG_DIR=/var/log/mtx.
mtx_notifier​_camel.log Notification Server messages are based on the sections of code that produce the message and the severity of the message:
  • debug
  • warn
  • error

Log settings are configured in /opt/mtx/conf/log4j2-mtx_notifier_camel.xml.

To customize Notification Server logging, copy /opt/mtx/data/log4j2-mtx_notifier_camel.xml to /opt/mtx/conf/log4j2-mtx_notifier_camel.xml and modify the file in the new location. Revert changes by copying from /opt/mtx/data again if necessary.

The default logging level is ERROR.

 Look for ERROR. For example:
2020.01.22:15:24:01:003:
[main]:ERROR:com.matrixx.camelNotifier.gateway
.RSGatewayServiceImpl:makeXmlCall
​has issue to communicate with Rsgateway for: 
http://localhost:8080/rsgateway/data/v3/mtx_config
org.apache.camel.CamelExecutionException: ​
Exception occurred during execution on the exchange: ​
Exchange[ID-hungle-laptop7-1579735438696-0-1]
rsgateway.log Contains messages for debugging the SubMan REST APIs.

Log settings are configured in /opt/mtx/conf/log4j2-rsgateway.xml.

To customize RS Gateway logging, copy /opt/mtx/data/log4j2-rsgateway.xml to /opt/mtx/conf/log4j2-rsgateway.xml and modify the file in the new location. Revert changes by copying from /opt/mtx/data again if necessary.

The location of the log file is specified by MTX_LOG_DIR, which is normally set as -DMTX_LOG_DIR=/var/log/mtx.

Look for "error." For example: 
2013.10.15:14:13:01:973:
[main]:ERROR:com.matrixx.
datacontainer.UpdateDataContainers:
Could not update data containers from 
a running engine
java.lang.RuntimeException: Exception 
caught sending message to engine

Add the following sub-elements to the <Loggers> element to enable debugging for the REST services.

  • Logging for requests and responses between the RS Gateway and the MATRIXX Engine: 
    <Logger name="com.matrixx.rsgateway.SendMessage.dump" 
level="error" additivity="false"/>
  • Logging for REST requests and responses between an RS Gateway client and the RS Gateway:
    <Logger name='com.matrixx.common.web.EntryExitFilter' 
level='trace'/>
Note: You must configure JSON and XML logging to view the incoming payload. The SendMessage.dump is the full MDC sent to the engine and the response in MDC dump format.
chf.log, nrf.log 5G network function messages are based on the sections of code that produce the message and the severity of the message:
  • debug
  • warn
  • error

For information about 5G logging configuration, see the 5G logging and alarms discussion in MATRIXX 5G Integration.
snmp-adapter.log SNMP Prometheus Adapter messages are based on the sections of code that produce the message and the severity of the message:
  • debug
  • warn
  • error

To customize SNMP Prometheus Adapter logging, copy /opt/mtx/data/log4j2-mtx-snmp-adapter.xml to /opt/mtx/conf/log4j2-mtx-snmp-adapter.xml and modify the file in the new location. Revert changes by copying from /opt/mtx/data again if necessary.
tra_util.log Logging information from Traffic Routing Agent (TRA) start/stop utilities. This information is also printed to stdout.
mtx_rcckp.log Logging information from the Route Cache Controller process handler.
resource_agent.log Logging information from OS-resource related commands, including logging information from VIP management functions.
http_price_controller.log Logging information from the HTTP Price Controller REST service (http_price_controller_app), which is responsible for retrieving an engine status, transmitting a pricing compilation to an engine, and loading a pricing compilation onto an engine. The log format is:

TimeStamp|LogLevel|Process ID|Thread ID|LogSource|Unique Tag Id / Log Id|LogMessage
Note: When objects are quarantined, messages are written to the mtx_debug.log file on the failed pod, the /var/log/messages file on the failed pod, and the mtx_debug.log file on the highest-numbered processing pod.

Transaction Log Files

MATRIXX Transaction Server stores transaction log files and the Global Transaction Counter (GTC) ranges they contain in these transaction log files on individual publishing servers:
  • The individual transaction log files in ${MTX_TXN_LOG_DIR}. For example, you might use the engine, cluster, and pod to name these directories: 
    ../local_1_2_1/blade_1_2_1/
    The transaction files in that directory have this naming syntax:
    transaction_engineNo_clusterNo_bladeNo_
timestamp_logNo.log.gz
    For example, this transaction log file stores transactions for engine 1, cluster 2, publishing server 1, and was created at the timestamp 1596948819:
    transaction_1_2_1_1596948819_1.log.gz
  • current_transaction_log_gtc_range — A temporary file created in ${MTX_TXN_LOG_DIR} that accumulates metadata about transaction log files and the GTC range each contains. Once the metadata file contains information for 1000 transaction log files, it is renamed with this syntax and moved to ${MTX_SHARED_DIR}:
    transaction_gtc_range_timestamp_lowerGTC_
upperGTC
    This example shows a transaction log metadata file created at the timestamp 1596948819, containing GTCs that range from 1 to 39821:
    transaction_log_gtc_range_1596948819_1_39821
    The transaction log file metadata entries in current_transaction_log_gtc_range have this syntax:
    lower_GTC, upperGTC ../${MTX_TXN_LOG_DIR}_path
/transaction_engine_no_cluster_no_blade_no_
timestamp_log_no.log.gz
    For example, the transaction_log_gtc_range_1596948819_1_39821 file might have these entries:
    1, 24, ../local_1_2_1/blade_1_2_1/transaction_1_2_1_1596948819_1.log.gz
25, 48, ../local_1_2_1/blade_1_2_1/transaction_1_2_1_1596948826_2.log.gz
49, 72, ../local_1_2_1/blade_1_2_1/transaction_1_2_1_1596948834_3.log.gz
73, 96, ../local_1_2_1/blade_1_2_1/transaction_1_2_1_1596948841_4.log.gz
97, 120, ../local_1_2_1/blade_1_2_1/transaction_1_2_1_1596948849_5.log.gz
...
32757, 39821, ../local_1_2_1/blade_1_2_1/transaction_1_2_1_1596948903_10.log.gz
Note: If the gtc_sorted_txn_logging_enabled parameter is enabled, the transaction log filename is in the format txn_<log_create_time><min_gtc><max_gtc>.log. Messages in MATRIXX transaction logs are listed in GTC order.

Java Log4j 2.x Files

You can switch Java-based MATRIXX components between different Log4j 2.x configuration files, and direct log output to different locations, by setting a component-specific variable in a file named after the component. You create the file with the variable setting in the /etc/default directory.

For example, to specify a /opt/mtx/data/log4j2-matrixx-test.xml file to configure logging for My MATRIXX different from the file in /opt/mtx/data/log4j2-matrixx.xml, create the /etc/default/mtx_mymatrixx file with the following contents (or similar):

MTX_MYMATRIXX_START_ARGS="--jvm_arg --log-config /opt/mtx/data/log4j2-matrixx-test.xml

Save and close the file, then restart My MATRIXX with the following command:

sudo systemctl restart mtx_mymatrixx

To specify a different location for log ouput, create the /etc/default/mtx_mymatrixx file with the following contents (or similar):

MTX_MYMATRIXX_START_ARGS="--jvm_arg -DMTX_LOG_DIR=full_path_to_directory

Where full_path_to_directory is the full path to the directory where log files are written when My MATRIXX is restarted.

For more information about setting Java options, see the discussion on setting Java options in MATRIXX Installation and Upgrade.