Complex Event Processing Tutorial

Complex Event Processing (CEP) has been called a "new computing paradigm" by some. This tutorial describes the motivation for this new technology, a brief overview of event processing, a description of the kinds of applications that have employed CEP, and an overview of the main technology elements of event processing.

The Motivation of CEP: the Need for a Better Way to Manage Event-Oriented Information

Events have existed in software architecture for years, but most software systems still use a "static" data processing model, where events are stored, indexed, and queried in attempt to figure out "what happened," "how much money was made last month," or, in a fraud detection context (shown at right), "what potentially fraudulent trades happened last week?"

Such static computing models were identified as insufficient by leading academics in the 90's (see the history of CEP here), because the static data processing model, which relies on relational databases, was felt to be fundamentally flawed. These researchers felt that, because databases are designed to answer historical questions, that they were incapable of properly processing events in the context of the now. This research examined ways of, rather than storing, indexing, and querying static "snapshots" of data, CEP research aimed to flip traditional computing models on their head, and examined ways to use in-memory techniques to insert queries in the stream of events, and to flows data over these queries, eliminating the overhead of the database.

The critical innovation of CEP was to extend traditional computing metaphors by building processing models and programming language techniques to concisely identify complex (hence the term "complex event processing") sequences of events: that is, that event A was followed by B, then C, with temporal constraints (within 5 hours).

What is an Event?

An event is a piece of data that represents that something happened in the real world, or in software system. Events flow in streams within every organization. Examples of events include:

Financial market events: “Buy 10,000 shares of IBM at $80.45.”
Supply chain events: “RFID tag 121.19.1818 was scanned at 1:39PM at dock door 10”
Security events: “TCP/IP address 128.1.32.298 just accessed server 5”

The Applications of CEP

Recently CEP has experienced rapid growth and adoption, and has publicly recognized in a variety of industries with documented use cases. Capital markets was the first market to heavily utilize CEP it forms the basis of algorithmic trading (tutorial on algorithmic trading) applications in top firms like Deutsche Bank, ABN Amro, and JP Morgan. Adoption of CEP has also occurred in compliance applications, fraud detection, airlines operations monitoring, and RFID-enabled supply chains. See the applications of CEP for more detail.

What is Event Processing?

Software vendors still use a variety of terms for event processing technologies, but most agree that event processing has four main technology elements: complex event processing (CEP), business activity monitoring (BAM), data stream management (DSM), and event middleware. Other popular terms for related technologies include event stream processing (ESP), and stream processing. The three core elements of event processing are:

Complex event processing is widely believed to be a key software technology to power more dynamic, real-time, automated business applications. CEP provides the basic language and execution engine for processing events. At the heart of CEP is an event-driven programming language that allows applications to concisely express important “complex” sequences of events (A followed by B, then C) with temporal (within 5 seconds) or spatial (within 5 miles) constraints. CEP languages provide the language capability to not only detect complex sequences, but to also control complex analysis and actions that often result.
Business activity monitoring (BAM) provides a graphical user interface for CEP. BAM "dashboards" differ from traditional graphical user interface (GUI) technologies in that they are designed to display and manipulate time-based information typically found in event processing systems.
Data stream management (DSM) is data management technology that manages event data. Also referred to as time-series database technology, DSM helps applications capture event state, perform root cause analysis on event data, and simulate anticipated automated conditions.
Event middleware is any form of connectivity that supplies events to or carries events from a CEP engine. Examples of event middleware include messaging systems such as JMS, publish-and-subscribe technology such as TIBCO's Rendezvous, or a messaging-based enterprise service bus, or ESB. Even a relational database can act as a source of, or a synch for, events.

Event Driven Architecture

Event driven architecture, or EDA, is the architecture of event processing technologies. Event processing provides the tools to build an EDA - programming, execution servers, BAM GUIs to display and managing events and event alerts. For a deeper discussion of EDA, SOA, and event processing, visit the EDA & SOA section.