Data Export

The MATRIXX Data Export Docker image (data-export) contains the data-export.jar utility which uses checkpoint files to transform database contents and MATRIXX Event File (MEF) files to comma-separated value (CSV) files. This image also creates SQL files for creating RDBMS tables and loading the data from the CSV files.

For information about Docker images, see the discussion about MATRIXX Digital Commerce Docker images and supporting files in MATRIXX Cloud Native Installation and Upgrade.

Creating RDBMS tables is a multi-step process that includes:
  • If necessary, running the create_checkpoint.py script on a publishing blade to create a new checkpoint of the in-memory database.
  • Running the data-export image to create CSV files used to load data into a relational database.
  • Running SQL scripts to create RDBMS tables and load the data.

See the discussion about third-party software requirements in Installation and Configuration for the supported SQL database release level.

Data Export

The data-export image takes MEFs and static checkpoints of the following databases as input:

  • Pricing
  • Balance set
  • Subscriber

Enter export parameters at the command line or use a configuration file to determine which MTX objects and fields to export. You can specify if the objects in the checkpoint and MEF files are transformed to SQL tables in the exported CSV files. The configuration file is a Java properties file with a .yaml file extension and its entries are stated in name:value format. Sample .yaml files are provided under the /opt/mtx/data directory, or you can create a custom configuration file.

If you have custom MDCs that extend MATRIXX MDCs, use the add_custom_field_to_export.py script to create a custom .yaml configuration file that includes your custom fields. For details, see the discussion about add_custom_field_to_export.py in MATRIXX Engine Integration.

Note: Array and list elements can be exported if the elements in the array or list are of simple types (not structs). The array or list is exported as a string with a comma as a delimiter between the elements.

Output Files

Based on the configurations in the .yaml file, the data-export image will generate the following files:

  • The extracted database content in one or more CSV files.
  • The create_tables.sql script, which creates new SQL tables for the exported data.
    Note: The properties files and the CREATE TABLE statements in the create_tables.sql script provide descriptions of the fields in the output CSV files.
  • The load_tables.sql script, which loads the exported CSV data into the SQL tables.
Note: The in-progress output CSV files have a temporary file extension, .tmp. The file extension is changed to .csv when the output file is complete.

Database Export and SQL Loading Process

The following steps describe the process for exporting subscription data or event data to CSV files and loading that data into new SQL tables:
  1. Select or create a configuration .yaml file to use with the data-export image based on the type of MTX objects you want to export (subscriber, event, or both).
    Tip: If you have custom MDCs that extend MATRIXX MDCs, use the add_custom_field_to_export.py script to create a new .yaml configuration file that includes your custom fields to use as input for the data-export image.
  2. Use the configuration .yaml file to run the data-export image and export your data.
  3. Execute the create_tables.sql script generated by the data-export image to create new SQL tables.
  4. Execute the load_tables.sql script generated by the data-export image to load the exported CSV data into the new SQL tables.