Event Streaming Overview

You use the MATRIXX Digital Commerce Event Streaming feature to produce custom event streams from the events that MATRIXX Digital Commerce produces. An event stream is an ordered, unbounded, and continuous flow of events. Monitor or analyze event data in real time using an external event-handling solution by creating event streams of specific events relevant to your networks and businesses.

As an event transaction processing platform, MATRIXX Digital Commerce produces events for a variety of reasons. Some events are single data records containing information about actions that occurred on the MATRIXX Engine cluster. If an event impacts multiple wallets, it becomes a primary event, and it is packaged with the secondary events for those other wallets. When primary and secondary EDRs are generated, they are always in the same MtxEventRecordData MDC (MATRIXX data container). For details on primary and secondary events, see the discussion about MATRIXX event detail records (EDRs) in MATRIXX Engine Integration.

Different events are derived from the MtxEvent object to represent different actions that occurred in the system. Some events, such as usage events, are produced frequently, while other events, such as those for payment processing and records of purchase, are produced less frequently.

MATRIXX Digital Commerce provides these components to support event streaming:
  • Event Stream Server — A MATRIXX process that generates, stores, and archives the events generated by the MATRIXX Engine, and if configured, streams the events through a gateway (the Event Streaming Framework ) to external event-handling solutions. The Event Stream Server runs on the publishing cluster of a MATRIXX Engine. For more information, see the discussion about Event Stream Server.

  • Event Streaming Framework — A gateway that connects a MATRIXX Engine cluster to external systems that consume the event streams. The current release of the Event Streaming Framework provides these connectors to stream events to event consumers:
    • Multi connector (a list of other connectors)
    • Apache Kafka
    • ActiveMQ
    • Google Pub/Sub
    • Disk (a file or directory)
    • Console (standard output)
    For more information, see the discussion about Event Streaming Framework connectors.
Note: The event streaming feature streams at least one copy of each event. Typically, only a single copy of each event is streamed. However duplicate events may be sent if the MATRIXX Engine encounters processing errors, restarts, fails over, or is processing very few transactions per second (TPS). The event consumer is responsible for taking any action appropriate for duplicate messages. Identify duplicate messages by checking whether they have identical EventId field values. For details on EventIds, see the discussion about MATRIXX Event Details Records in Architecture Overview.

Figure 1 shows the main event streaming components in a minimal deployment. Internally MATRIXX Digital Commerce interprets event streams as transaction streams.

Figure 1. Event Streaming Architecture


For more detail on the event streaming data formats used by MATRIXX Digital Commerce, see the discussion about MATRIXX Engine data model.

The MATRIXX Engine publishing cluster includes two running publishing blades, one active and one hot standby, and a checkpointing blade. The publishing cluster and checkpointing blades perform these tasks:
  • The checkpointing blade creates database checkpoints.
  • The active publishing blade:
    • Processes the transaction stream sent by the processing cluster.
    • Creates transaction log files from transaction logs.
    • Moves the transaction logs to the archive directory.
    • Creates MATRIXX Event File (MEF).
    • Sends the transaction stream to the processing cluster on the MATRIXX standby engine, which in turn sends it to the active processing blade on next MATRIXX standby engine (if any).
  • The standby publishing blade participates in replay transaction processing.
When the Event Loader is enabled, the active publishing blade loads events into the Event Repository. If event streaming is enabled, the active publishing blade publishes stream events to the Event Streaming Framework.

Both publishing blades (active and standby) create internal transaction log files that are used for fast failover. For information about recovering event stream information in the case of a system outage or engine failover, see the discussions about recovering event stream information, and MATRIXX transaction logs.

For a summary of tasks required to implement event streaming in a MATRIXX Digital Commerce environment, see the discussion about implementing event streaming.

To display statistics about event stream information processed by the Event Stream Server on the engine, see the discussion about monitoring event streams.

To reconfigure an event stream to change the type of data streamed during runtime, see the discussion about reconfiguring event streaming during runtime.

Event Streaming Framework HA Features

Use the MATRIXX Event Streaming Framework HA features if realtime event streaming is critical to your business. The HA features ensure pauseless realtime event streaming using hardware and software redundancy. If you expose event streams to customers, MATRIXX strongly recommends that you use the HA features.

A minimal Event Streaming deployment includes a single Event Streaming Framework server running on the MATRIXX Gateway Proxy on a single physical server. To form an Event Streaming Framework HA group, install proxy server RPMs on different physical servers and configure them as Event Streaming Frameworks. They form an HA cluster and share the streaming and sub-domain configuration settings. You do not need to run the other MATRIXX Proxy Server components on these servers, and external connectivity is unnecessary.