Data Export Environment Variables
Configure the following environment variables for containers created from the data-export image.
The Data Export image extends from the MATRIXX Java image and inherits the features that it implements (such as configuration synchronization from /sync, JVM arguments, debugging, and Flight Recorder options). See the discussion about using containers in MATRIXX Configuration for more information.
Environment Variables Specific to Data Export describes the environment variables that configure Data Export.
| Environment Variable Name | Description | Default Value |
|---|---|---|
| MTX_CHECKPOINTS_DIRECTORY | The directory with the checkpoint directories. The image selects the latest directory matching 'mtx_ckpt_' from this directory as the input directory. If you specify an MTX_INPUT_DIRECTORY value, Data Export does not start the check for the latest checkpoint directory. | /shared/checkpoints |
| MTX_INPUT_DIRECTORY | The checkpoint directory used as the input. If specified, the value of MTX_CHECKPOINTS_DIRECTORY is ignored. | |
| MTX_OUTPUT_DIRECTORY | The directory where the output of the data export is written. The image creates this directory if it does not exist. | /opt/mtx/data/export |
| MTX_OUTPUT_DIRECTORY_POSTFIX_EXP | This is a bash expression that is evaluated at runtime to create
the output directory postfix. For
example: This creates a timestamp directory for the output. For example:/opt/mtx/data/export/20210121-104635 |
|
| MTX_CUST_MDC_CONFIG_FILE | The path to the custom MDC configuration file. | /opt/mtx/conf/mdc_config_custom.xml |
| MTX_VERSION | This is the MATRIXX Version to pass to the Data Export command. | |
| MTX_EXPORT_CONFIG_FILE | The export configuration file to pass to the data export command.
The image has the following default configurations in
/opt/mtx/conf/:
|
/opt/mtx/conf/subscriber_export.yaml |
| MTX_SQL_SCHEMA_NAME | An optional value that specifies the SQL schema file to pass to the Data Export command. | |
| MTX_DATA_LOG_LEVEL | The logging level to use. This can be critical,
error, warn,
info, debug,
trace, or off. |
info |
| MTX_LOG_MECHANISM | The mechanism used for logging output. By default, logs are sent to STDOUT. Set this to FILE to log to /var/log/mtx/data_export.log. | STDOUT |
Running Data Export
The following example shows how to use the Data Export image using Docker. Checkpoints are written to a directory called /opt/mtx/shared on the image host machine. Data Export writes output to the /opt/mtx/data/export/<timestamp> on the host machine.
docker run --rm \
-v /opt/mtx/shared:/shared \
-v /opt/mtx/data/export:/opt/mtx/data/export \
-e MTX_VERSION=5212 \
-e MTX_OUTPUT_DIRECTORY_POSTFIX_EXP="echo "/$(date +"%Y%m%d-%H%M%S")"" \
mydockerregistry/data-export:5212The following example shows the output from the command.
-- Docker Image: matrixx-java Version: 5212 Build Number: 79830 Built: 2021-01-21 10:44:06 UTC --
Java Runtime:
openjdk version "11.0.9.1" 2020-11-04 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.9.1+1-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.9.1+1-LTS, mixed mode, sharing)
-- Docker Image: data-export Version: 5212 Build Number: 79830 Built: 2021-01-21 10:46:28 UTC --
Logging to STDOUT
Synchronising User Configuration into /opt/mtx ...
sending incremental file list
sent 112 bytes received 17 bytes 258.00 bytes/sec
total size is 0 speedup is 0.00
Synchronising of User Configuration Complete
Running /opt/mtx/bin/docker-entrypoint.sh with [] ....
The latest Checkpoint directory is '/shared/checkpoints/mtx_ckpt_001'
Output Directory Postfix is: /20210121-112053
Updating the output directory to /opt/mtx/data/export/20210121-112053
Input Directory: /shared/checkpoints/mtx_ckpt_001
The output directory '/opt/mtx/data/export/20210121-112053' does not exist. Attempting to create it
mkdir: created directory '/opt/mtx/data/export/20210121-112053'
Custom MDC Configuration File: /opt/mtx/conf/mdc_config_custom.xml
MATRIXX Version: 5212
Export Configuration File: /opt/mtx/conf/subscriber_export.yaml
SQL Schema Name:
Using Override Log4j configuration file at /opt/mtx/conf/log4j2-mtx_data_export.xml
2021-01-21 11:20:53 [INFO] Starting Data Exporter Process...
pricing loaded into memory
0 subscriber_checkpoint and MEF files processed
create_tables.sql and load_tables.sql files created