Home
MATRIXX 5G Event Streaming
Appendixes
ASN.1 sFTP Sink Record Recovery Modes
Record recovery has four modes: MOD_RANGE, MOD_RANGE_GLOBAL, MOD_CONTINUE, and MOD_CONTINUE_GLOBAL. 5g_integration/r_5g_events_rr_modes.html#concept_nx3_sdg_s5b__table_cjf_lgg_s5b describes the modes in two groups.
MOD_RANGE Record Recovery Mode
In MOD_RANGE mode, record recovery accepts input of a list of metadata files. It generates a recovery file for each metadata file. 5g_integration/r_5g_events_rr_mod_range.html#concept_o5w_nkg_s5b__table_w44_v3t_s5b describes the configuration property used to specify a range of metadata files by filename.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
- About MATRIXX 5G Event Streaming
  MATRIXX 5G Event Streaming describes how to aggregate and stream charging data records (CDRs) from the SBA Gateway to an sFTP server using a Kafka broker.
- 5G Event Streaming Overview
  SBA Gateway can stream HTTP requests and MATRIXX Engine HTTP responses to a Kafka broker. The 5G event streaming applications convert the received JSON objects to ASN.1 binary format for export to Kafka applications and can optionally aggregate and route based on content. The GTP Prime Adapter microservice is provided for sending records from Kafka to a GPRS Tunneling Protocol (GTP) server.
- Converting CDRs to ASN.1 Format
  Charging data records (CDRs) are processed by the CDR ASN.1 Streamer Kafka client application to convert them from JSON into ASN.1 objects. After the ASN.1 objects are serialized as a binary message using Basic Encoding Rules (BER), they are placed in another Kafka topic for further processing. The ASN.1 sFTP Sink application then writes the message to file before transferring it over Secure FTP (sFTP).
- sFTP Sink Microservice Deployment
- ASN.1 Record Naming and Counting
  Each time the ASN.1 sFTP Sink application receives a batch of records, if the number of records is greater than or equal to the configured closure condition, the aggregated ASN.1 record is generated. Otherwise, the batch is held for the next ASN.1 record.
- CDR Aggregation Function
  Every request received by SBA Gateway from the SMF results in the creation of a 5G event, also known as a network _glossary/cdr.html. To avoid sending excessive records to a downstream system, CDRs must be aggregated on a per session basis so that only summary level CDRs are sent to third-party CGF or the BSS domain. The CDR Aggregation Function (CAF) application is a Kafka Streams application that aggregates network interactions handled by SBA Gateway on the 5G N40 interface.
- Content-Based CDR Routing
  The CDR Kafka router republishes each _glossary/cdr.html based on its content, either as received or enriched with additional data on a conditional basis.
- Write To Disk
  When Kafka is unavailable, the producer continues attempting to send to the Kafka broker until a timeout is reached. On timeout, the records are written to a separate log file on disk.
- Record Formats and Schemas
  Request and response records are sent in JSON format.
- Kafka Partition Key
  The Kafka partition key used is the SessionId for the exchange. If none is available, for example if there is an error returned by the engine, then no partition key is used.
- Installing 5G Event Streaming Applications
  The 5G event streaming applications are delivered as images for container creation in a Kubernetes cluster using a Helm chart. The following sections describe the installation process. The procedures assume you have knowledge of and access to the kubectl tool.
- Configuring 5G Event Streaming
  Configure 5G event streaming by providing values for properties in one or more Helm values files.
- Configuring the Applications
  The 5G event streaming applications are configured in the Helm values file, the nf.yaml file, and an application.yaml files for each application.
- Applying Configuration
  You can apply configuration for 5G event streaming applications in several ways using the Helm values chart. These methods are not mutually exclusive and can be used together.
- Appendixes
  - ASN.1 sFTP Sink Record Recovery Modes
    Record recovery has four modes: MOD_RANGE, MOD_RANGE_GLOBAL, MOD_CONTINUE, and MOD_CONTINUE_GLOBAL. 5g_integration/r_5g_events_rr_modes.html#concept_nx3_sdg_s5b__table_cjf_lgg_s5b describes the modes in two groups.
    - MOD_RANGE Record Recovery Mode
      In MOD_RANGE mode, record recovery accepts input of a list of metadata files. It generates a recovery file for each metadata file. 5g_integration/r_5g_events_rr_mod_range.html#concept_o5w_nkg_s5b__table_w44_v3t_s5b describes the configuration property used to specify a range of metadata files by filename.
    - MOD_RANGE_GLOBAL Record Recovery Mode
      MOD_RANGE_GLOBAL mode accepts input of global start and end timestamps for all partitions and generates recovery files for all records of each partition. Configure the file number by specifying the node ID, file sequence number (FSN), and running count/file issued number (RC).
    - MOD_CONTINUE Record Recovery Mode
      In MOD_CONTINUE mode, record recovery accepts input of a list of partition offsets or time stamps. After configuration is applied, production pods are reset to the offset or time stamp and files are generated as normal.
    - MOD_CONTINUE_GLOBAL Record Recovery Mode
      MOD_CONTINUE_GLOBAL mode accepts input of a global start time stamp. After the configuration is applied, production pods are reset to the offset or time stamp and files are generated as normal.
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

MOD_RANGE Record Recovery Mode

In MOD_RANGE mode, record recovery accepts input of a list of metadata files. It generates a recovery file for each metadata file. MOD_RANGE Configuration Properties describes the configuration property used to specify a range of metadata files by filename.

Table 1. MOD_RANGE Configuration Properties
Property	Description
sftp.internal.recovery.modRange.metadataFiles	List of metadata files, by filename.

This mode runs as a Kubernetes job specified in asn1-kafka-sftp-recovery.yaml. Configuration is specified in the ConfigMap section. The following asn1-kafka-sftp-recovery.yaml example is configured for MOD_RANGE mode, recovering the records described in the asn1-sftp-sink-sink-0_-_1.20220811_._1822+0000.json and asn1-sftp-sink-sink-0_-_3.20220811_._1822+0000.json metadata files.

kind: ConfigMap
apiVersion: v1
metadata:
  name: sink-recovery-config
  namespace: matrixx
data:
  asn1-sftp-sink.yaml: |-
    kafka:
      configuration:
        bootstrap.servers: mtx-kafka-cp-kafka.confluent.svc.cluster.local:9092
    sftp:
      remote:
        user: mtxsftpuser
        password: mtx123
        host: sftp.sftp.svc.cluster.local
        port: 22
      internal:
        buffer:
          path: "/opt/mtx/recovery/buffer"
        uploader:
          path: "/opt/mtx/recovery/internal_recovery"
             
        recovery:
          enabled: true
          path: "/opt/mtx/recovery"
          modRange:
              metadataFiles: [asn1-sftp-sink-sink-0_-_1.20220811_._1822+0000.json,asn1-sftp-sink-sink-0_-_3.20220811_._1822+0000.json]
      routes:
        - topic: cdr-asn1-1
          path: "upload/uploads-1"
          isdefault: true
        - topic: cdr-asn1-2
          path: "upload/uploads-2"

MOD_RANGE mode does the following:

Retrieves specified metadata files.
Collects Kafka starting and ending offsets from each metadata file.
Locates records using the specified Kafka information using the seek function and reprocesses them.
When all records described by the metadata files are collected, the ASN.1 record is generated. The name of the record is generated using the file naming configuration provided and the values inside the metadata file. For more information, see the discussion about file naming configuration.
The job stops.