Configure Event Restreaming for Failed Events

Perform this task to restream failed events.

About this task

Events are failed if they cannot be published to the event consumer. To restream those failed events, stop the Event Streaming Framework pods, restream the failed events from a starting cursor to an ending cursor, and then restart all the pods.

Note: Another way to do this is to first generate or manually create a Streamed Event File (SEF) from transaction log files and then restream a Global Transaction Counter (GTC) range. Manually created SEFs must be placed into the stream_events tree. A new SEF directory can be created for the new SEFs with a matching minimum and maximum GTC. For example, for the set of log files: /tmp/txnlogs/blade_12_2_1/transaction_12_2_1_1683845620_859_73449_73452.log.gz, run this command to produce the SEF file:

create_event_files_from_txn_logs -d -t -l /tmp 73449 73452

For more information, see the discussion about the create_event_files_from_txn_logs script.

Procedure

Stop the Event Streaming Framework.

Update the event-streamer-restream.yaml file to look as follows:

apiVersion: batch/v1
kind: Job
metadata:
  name: event-streamer-restream
  namespace: matrixx-s2tra
spec:
  template:
    spec:
 
      initContainers:
      - args:
        env:
        - name: TZ
          value: UTC
        image: a.j.m0012242008.com/mtx-docker-eng-snapshot/matrixx-toolbox:5252
        imagePullPolicy: Always
        name: rsgateway-prerequisite-check
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - env:
        - name: MTX_SYNC_TARGET_DIRS
          value: /opt/mtx
        - name: TZ
          value: UTC
        image: a.j.m0012242008.com/mtx-docker-eng-snapshot/mtx-config:5250.evs.topo
        imagePullPolicy: Always
        name: sideloader-mtx-config-0
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sync
          name: sideloader-sync-dir
          
      nodeName: mtx-huawei2-bld05
      volumes:
      - emptyDir: {}
        name: sideloader-sync-dir
      - name: shared-logging-storage
        persistentVolumeClaim:
          claimName: shared-logging-storage
      securityContext:
        fsGroup: 1000
      serviceAccount: default
      serviceAccountName: default
      imagePullSecrets:
      - name: matrixx-docker-registry-creds
      containers:
      - args:
        - --no-colors
        - --args
        - -restream -startgtc:57000001 -endgtc:57500001 -props:/opt/mtx/conf/mtx_event_streamer.yaml
        env:
        - name: MTX_STRM_LOG_LEVEL
          value: info
        - name: MTX_RSGW_HOST
          value: rsgateway-ag1
        - name: MTX_RSGW_PORT
          value: "8080"
        - name: MTX_LOG_MECHANISM
          value: FILE
        - name: TZ
          value: UTC
        image: a.j.m0012242008.com/mtx-docker-eng-snapshot/event-streamer:5252
        imagePullPolicy: Always
        name: event-streamer-restream
        ports:
        - containerPort: 5765
          name: http
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - check_event_streamer.sh
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sync
          name: sideloader-sync-dir
        - mountPath: /var/log/mtx
          name: shared-logging-storage
      restartPolicy: Never

Run these Kubernetes commands to set the replicas to 0 (zero), apply configuration to enable streaming mode, and restart one or more replicas:
```
kubectl --namespace yournamespace scale --replicas=0 deployment/event-streamer-release
kubectl --namespace yournamespace apply -f event-streamer-restream.yaml
kubectl --namespace yournamespace scale --replicas=1 deployment/event-streamer-release
```
Where:
- yournamespace is the namespace you used when installing MATRIXX, for example, matrixx.
- deployment/event-streamer-release is a deployment of the event streamer within Kubernetes, created by installing the MATRIXX Helm chart with the event streamer enabled. This command scales the number of instances (to zero with the first command, and back to 1 with the third command).
- release is the release name in the MATRIXX Helm chart (ag1 by default).