Home
MATRIXX 5G Event Streaming
Write To Disk
When Kafka is unavailable, the producer continues attempting to send to the Kafka broker until a timeout is reached. On timeout, the records are written to a separate log file on disk.

Welcome
MATRIXX Release Notes
MATRIXX Architecture
MATRIXX Installation and Upgrade
MATRIXX Configuration
MATRIXX Security
MATRIXX Integration
MATRIXX Diameter Integration
MATRIXX Call Control Framework Integration
MATRIXX TM Forum Integration
MATRIXX 5G Integration
MATRIXX 5G Event Streaming
- About MATRIXX 5G Event Streaming
  MATRIXX 5G Event Streaming describes how to aggregate and stream charging data records (CDRs) from the SBA Gateway to an sFTP server using a Kafka broker.
- 5G Event Streaming Overview
  SBA Gateway can stream HTTP requests and MATRIXX Engine HTTP responses to a Kafka broker. The 5G event streaming applications convert the received JSON objects to ASN.1 binary format for export to Kafka applications and can optionally aggregate and route based on content. The GTP Prime Adapter microservice is provided for sending records from Kafka to a GPRS Tunneling Protocol (GTP) server.
- Converting CDRs to ASN.1 Format
  Charging data records (CDRs) are processed by the CDR ASN.1 Streamer Kafka client application to convert them from JSON into ASN.1 objects. After the ASN.1 objects are serialized as a binary message using Basic Encoding Rules (BER), they are placed in another Kafka topic for further processing. The ASN.1 sFTP Sink application then writes the message to file before transferring it over Secure FTP (sFTP).
- sFTP Sink Microservice Deployment
- ASN.1 Record Naming and Counting
  Each time the ASN.1 sFTP Sink application receives a batch of records, if the number of records is greater than or equal to the configured closure condition, the aggregated ASN.1 record is generated. Otherwise, the batch is held for the next ASN.1 record.
- CDR Aggregation Function
  Every request received by SBA Gateway from the SMF results in the creation of a 5G event, also known as a network _glossary/cdr.html. To avoid sending excessive records to a downstream system, CDRs must be aggregated on a per session basis so that only summary level CDRs are sent to third-party CGF or the BSS domain. The CDR Aggregation Function (CAF) application is a Kafka Streams application that aggregates network interactions handled by SBA Gateway on the 5G N40 interface.
- Content-Based CDR Routing
  The CDR Kafka router republishes each _glossary/cdr.html based on its content, either as received or enriched with additional data on a conditional basis.
- Write To Disk
  When Kafka is unavailable, the producer continues attempting to send to the Kafka broker until a timeout is reached. On timeout, the records are written to a separate log file on disk.
  - Logging Behavior
    5g_integration/c_wtd_log_behavior.html#concept_hgc_wny_hyb__table_a5s_cpy_hyb describes the variables that control aspects of Write to Disk log4j2 CDR logging such as the file path, name, and rollover behavior.
  - Write to Disk Replay
    The cdr-kafka-replay microservice can be configured to read log files from a specific directory by setting replay.directory appropriately. The records are sent to Kafka. If any fail, the process backs off for a period of time before retrying. After all records are successfully sent to Kafka, the job terminates.
  - Persistent Volumes
    Write to Disk uses a persistent volume (PV) shared by all instances of SBA Gateway using persistent volume claims (PVCs).
- Record Formats and Schemas
  Request and response records are sent in JSON format.
- Kafka Partition Key
  The Kafka partition key used is the SessionId for the exchange. If none is available, for example if there is an error returned by the engine, then no partition key is used.
- Installing 5G Event Streaming Applications
  The 5G event streaming applications are delivered as images for container creation in a Kubernetes cluster using a Helm chart. The following sections describe the installation process. The procedures assume you have knowledge of and access to the kubectl tool.
- Configuring 5G Event Streaming
  Configure 5G event streaming by providing values for properties in one or more Helm values files.
- Configuring the Applications
  The 5G event streaming applications are configured in the Helm values file, the nf.yaml file, and an application.yaml files for each application.
- Applying Configuration
  You can apply configuration for 5G event streaming applications in several ways using the Helm values chart. These methods are not mutually exclusive and can be used together.
- Appendixes
MATRIXX Event Streaming
MATRIXX Administration
MATRIXX Web App Administration
MATRIXX Monitoring and Logging
MATRIXX Policy
MATRIXX Kafka CDR Consumer
MATRIXX Pricing and Rating
MATRIXX Pay Now
MATRIXX Subscriber Management
MATRIXX Subscriber Management API
MATRIXX Business API SDK
My MATRIXX Help
MATRIXX Backoffice Customer Tool Help
MATRIXX Third-Party Licenses
Glossary

Write To Disk

When Kafka is unavailable, the producer continues attempting to send to the Kafka broker until a timeout is reached. On timeout, the records are written to a separate log file on disk.

When gateway.external.kafka.enabled is set to true:

SBA Gateway attempts to send to Kafka. If it is unsuccessful, SBA Gateway increments the metric: com.matrixx.cdr.producer.failed.total.
A warning of the failure cause is logged. The charging data record (CDR) itself is not added to the logs.

To write records to logs, set gateway.external.kafka.enabled and gateway.external.kafka.logger.enabled to true. When those are set:

SBA Gateway attempts to send to Kafka. If it is unsuccessful, SBA Gateway increments metric: com.matrixx.cdr.producer.failed.total.
A warning of the failure cause is logged.
The CDR itself is added to the logs.

Note: When the Write to Disk feature is enabled, MATRIXX Support strongly recommends that gateway.external.kafka.writeQueueMaxSize be set to an appropriate nonzero value. The Write to Disk feature and a nonzero setting for this property ensure that CDRs which cannot be written to Kafka are written to disk instead, with a threshold set to prevent out-of-memory issues if Kafka is unavailable.

The com_matrixx_cdr_producer_pending metric shows the number of outstanding network CDRs that have not yet been written to Kafka or to disk. The gateway.external.kafka.writeQueueMaxSize property sets a threshold on this value. Because the pending queue is a metric, it requires metrics to be enabled (they are enabled by default). If gateway.external.kafka.writeQueueMaxSize is enabled, do not disable metrics (do not set gateway.external.kafka.metrics.emabled or gateway.monitoring.prometheus.enabled to false).