Event Streaming Overview
You use the MATRIXX Digital Commerce Event Streaming feature to produce custom event streams from the events that MATRIXX Digital Commerce produces. An event stream is an ordered, unbounded, and continuous flow of events. Monitor or analyze event data in real time using an external event-handling solution by creating event streams of specific events relevant to your networks and businesses.
As an event transaction processing
platform, MATRIXX Digital Commerce produces events for a variety of reasons. Some events are single data records
containing information about actions that occurred on the MATRIXX Engine cluster. If an event impacts multiple wallets, it becomes a primary event, and it is
packaged with the secondary events for those other wallets. When primary and secondary
EDRs are
generated, they are always in the same MtxEventRecordData
MDC (MATRIXX data container). For
details on primary and secondary events, see the discussion about MATRIXX event detail
records (EDRs) in MATRIXX Engine Integration.
Different events are derived from the MtxEvent object to represent different actions that occurred in the system. Some events, such as usage events, are produced frequently, while other events, such as those for payment processing and records of purchase, are produced less frequently.
- Event Stream Server — A MATRIXX process that generates, stores, and archives the events generated by the MATRIXX Engine, and if configured, streams the events through a gateway (the Event Streaming Framework ) to external event-handling solutions. The Event Stream Server runs on the publishing cluster of a MATRIXX Engine. For more information, see the discussion about Event Stream Server.
- Event Streaming Framework — A gateway that connects a MATRIXX Engine cluster to external systems that consume the event streams. The current
release of the Event Streaming Framework provides these connectors to stream events to event consumers:
- Multi connector (a list of other connectors)
- Apache Kafka
- ActiveMQ
- Google Pub/Sub
- Disk (a file or directory)
- Console (standard output)
EventId
field values. For details on
EventId
s, see the discussion about MATRIXX Event Details Records in
Architecture Overview.Figure 1 shows the main event streaming components in a minimal deployment. Internally MATRIXX Digital Commerce interprets event streams as transaction streams.
For more detail on the event streaming data formats used by MATRIXX Digital Commerce, see the discussion about MATRIXX Engine data model.
- The checkpointing blade creates database checkpoints.
- The active publishing blade:
- Processes the transaction stream sent by the processing cluster.
- Creates transaction log files from transaction logs.
- Moves the transaction logs to the archive directory.
- Creates MATRIXX Event File (MEF).
- Sends the transaction stream to the processing cluster on the MATRIXX standby engine, which in turn sends it to the active processing blade on next MATRIXX standby engine (if any).
- The standby publishing blade participates in replay transaction processing.
Both publishing blades (active and standby) create internal transaction log files that are used for fast failover. For information about recovering event stream information in the case of a system outage or engine failover, see the discussions about recovering event stream information, and MATRIXX transaction logs.
For a summary of tasks required to implement event streaming in a MATRIXX Digital Commerce environment, see the discussion about implementing event streaming.
To display statistics about event stream information processed by the Event Stream Server on the engine, see the discussion about monitoring event streams.
To reconfigure an event stream to change the type of data streamed during runtime, see the discussion about reconfiguring event streaming during runtime.
Event Streaming Framework HA Features
Use the MATRIXX Event Streaming Framework HA features if realtime event streaming is critical to your business. The HA features ensure pauseless realtime event streaming using hardware and software redundancy. If you expose event streams to customers, MATRIXX strongly recommends that you use the HA features.
A minimal Event Streaming deployment includes a single Event Streaming Framework server running on the MATRIXX Gateway Proxy on a single physical server. To form an Event Streaming Framework HA group, install proxy server RPMs on different physical servers and configure them as Event Streaming Frameworks. They form an HA cluster and share the streaming and sub-domain configuration settings. You do not need to run the other MATRIXX Proxy Server components on these servers, and external connectivity is unnecessary.