Daily MEF Publishing Report

A daily report file is published for MATRIXX Event Files (MEFs) including event counts and Global Transaction Counters (GTCs).

You can use this report to troubleshoot for missing MEFs and then reprocess them when some issue affects MEF creation from GTCs. The publish_mefs.py script is run on the publishing pod after each batch of MEF files is synced with the publishing target. An example of a MEF file is mef_Time_minGtc_maxGtc.gz. The publish_mefs.py script detects MEF files on a publishing target with a modification time on a report date.

MEF daily reports depend on modification time of meta files at the pub target. If the modification time of a meta file is changed to a different date, the matching MEF file is listed in a MEF report for that date. The publish_mefs.py script uses meta files whose names begin with lst, for example, lst_minGtc_maxGtc_Time.txt, to detect the event count in a MEF file, where:

  • minGtc is the minimum GTC, for example, 1, in the range of MEF files in the meta file.
  • maxGtc is the maximum GTC, for example, 6183, in the range of MEF files in the meta file.
  • Time is the UTC time of the first file creation, for example, 168153834, in the meta file. This time is used to purge the meta files after two weeks.
    Note: Report files are not purged. Only meta files are purged.

An example filename for a meta file is lst_1_6183_1681538341.txt.

The following is an example of meta file contents:
1,1033,1000,mef_1681538286_1_1033.xml.gz
1034,2033,1000,mef_1681538286_1034_2033.xml.gz
2037,4533,1000,mef_1681538286_2037_4533.xml.gz
999,5033,1000,mef_1681538286_999_5033.xml.gz
5034,6051,586,mef_1681538286_5034_6051.xml.gz
6052,6183,80,mef_1681538286_6052_6183.xml.gz
Each line in a meta file has a minimum GTC, maximum GTC, event count, and MEF filename. The meta file is published before MEF files are sent to the publishing target. This ensures that each published MEF file has an entry in a meta file on the publishing target. The publish_mefs.py script reads the current report file and appends information about MEF files that are not listed yet in the report file. Only successfully published MEF files are put into the report.

An example of a MEF report filename for the current day is rep_1681455600_14Apr2023. The first number in the filename is the local UTC midnight time, and the second number is the same time in a human-readable format. One report is created each day including information from midnight to midnight in the local time zone. On the next day, the filename prefix is changed from rep to cmr.

Important: The date/time on the publishing pod must be the same as the date/time in the remote publishing pod for the MEF report to be published correctly.

The MEF report is updated only if new MEF files are published since the last run of the script. Files are sorted on the MEF report by minimum GTC.

You can detect GTC gaps and overlap errors in MEF files for a particular day by reading a list of MEF filenames from the corresponding report file and examining the minimum and maximum GTCs.