Write to Disk Replay

The cdr-kafka-replay microservice can be configured to read log files from a specific directory by setting replay.directory appropriately. The records are sent to Kafka. If any fail, the process backs off for a period of time before retrying. After all records are successfully sent to Kafka, the job terminates.

Replay has the following requirements:

  • Log files must be static and in the process of being updated by SBA Gateway.
  • Log files can be compressed and uncompressed. (Only the .gz extension is supported for compressed files.)
  • The files are read and processed alphabetically.
  • The files are not deleted. The corresponding cdr-kafka-replay logs report the last successful file and record the line number for use in any subsequent replay attempts.
Note: The value of the replay.directory property, /var/events/records/, matches the container mountPath.

Replay Configuration Properties describes the properties available for configuring the cdr-kafka-replay job.

Table 1. Replay Configuration Properties
Property Description
replay.kafka.config.bootstrap.servers Specifies Kafka broker addresses as a comma-separated list of URLs. For example: 
kafka1:9092,kafka2:9093
replay.topic The Kafka topic name of the destination topic for replayed records. For example, cdr.
replay.directory The absolute path (posix-compliant) to the location where log files are mounted. For example: 
/var/events/records
Log files must be in a single directory.
replay.kafka.config.* (Optional) Kafka producer client configuration overrides, for example, for setting SSL options.
replay.batchSize (Optional) The number of records to send as a batch.
replay.excludeExtensions (Optional) Files with the specified extensions are ignored.
replay.kafka.connectionMaxAttempts (Optional) The maximum times to retry when the broker URL is unresolvable when creating the Kafka producer on start-up.
replay.kafka.connectionRetrySeconds (Optional) The amount of time in seconds between connection retries.
replay.replayFrom.file (Optional) The file to resume from.
replay.replayFrom.line (Optional) The line number in the specified file to resume from.
replay.metrics.pushgateway.enabled (Optional) When set to true, Prometheus metrics are pushed to an external Prometheus Pushgateway. The default value is false.
replay.metrics.pushgateway.port (Optional) The Pushgateway port. The default value is 9091.
replay.metrics.pushgateway.host (Optional) The Pushgateway host. The default value is pushgateway.
replay.metrics.pushgateway.uriBase (Optional) The Pushgateway base URI, before labels. The default value is /metrics/job/cdr_replay.
replay.metrics.pushgateway.labels (Optional) A list of labels to add to the Pushgateway API URL. The default list is:
labels:
  - name: application
    value: cdr-replay
  - name: instance
    valueFromEnv: HOSTNAME
    base64Encoded: true

In normal operation, cdr-kafka-replay replays each file it finds in alphabetical order to accommodate rolled log files. When a record is successfully sent to Kafka, the filename and line number is logged. In the following example, the record on line 2 in file mtx-cdr-1.log has been successfully sent to Kafka:

2023-08-01 08:03:02.260 |  INFO |  | vert.x-eventloop-thread-0 | com.matrixx.events.replay.Replay | Successfully sent mtx-cdr-1.log:2

When records are rejected by Kafka, the job terminates. The last logged file and successful record can be used to restart the job. To resume from that point forward, set the configuration options replay.replayFrom.file and replay.replayFrom.line. If the line shown in the example indicates the last successful record to be sent, resumption replay would require the following configuration:

replay:
    replayFrom:
        file: mtx-cdr-1.log
        line: 3

Line numbers indicate the line number in the log file, starting from 1. As the record on line 2 was the last successful replay logged, it is correct to continue from line 3.

Pushgateway Configuration

When enabled, the Replay job pushes the current Prometheus metrics to an external Pushgateway. The CDR Replay configuration allows additional labels to be customized with both fixed values and values derived from environment variables local to the CDR Replay container. The following are (default) examples of labels:

labels:
  - name: application         # a name is always required
    value: cdr-replay         # a static value can be specified, if it is not RFC 4648 compliant, set base64Encoded to true (as per next)
 
  - name: instance           
    valueFromEnv: HOSTNAME    # the value is taken from the container's environment variables
    base64Encoded: true       # the value is further encoded in base64

The label configuration is used to add the request URI for the Pushgateway API.

Note: Providing any list in replay.metrics.pushgateway.labels replaces the defaults.

The cdr-kafka-replay job runs the microservice. After all records have been sent to Kafka or an error has occurred, the microservice shuts down and the job terminates. Removal of terminated jobs and their corresponding pod resources must be explicitly managed. The following is an example of a Kubernetes manifest that sets up a ConfigMap with configuration to run the replay job:

#------------- CDR REPLAY JOB ----------------
#
kind: ConfigMap
apiVersion: v1
metadata:
  name: cdr-replay-config
  namespace: matrixx
data:
  cdr-replay.yaml: |-
    replay:
      kafka:
        config:
            bootstrap.servers: kafka:9092      # Example Broker
      topic: cdr                               # CDR topic
      directory: /var/events/records           # Directory mounted
  
---
kind: Job
apiVersion: batch/v1
metadata:
  name: cdr-replay-job
  namespace: matrixx
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: cdr-replay
    spec:
      restartPolicy: Never
      volumes:
        - name: cdr-replay-config-vol
          configMap:
            name: cdr-replay-config
        - name: shared-events-replay-storage
          persistentVolumeClaim:
            claimName: sba-5gc-chf-shared-networkcdr-storage
      containers:
        - name: replay
          image: a.j.m0012242008.com/mtx-docker-eng-release-local/cdr-kafka-replay:1.0.2        # UPDATE AS APPROPRIATE
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /sync/conf
              name: cdr-replay-config-vol
            - name: shared-events-replay-storage
              mountPath: /var/events/records