Stream processing

Stream processing is a computational model in which data is treated as a continuously arriving sequence of records rather than as a finite collection available before execution begins. A stream processor applies transformations, maintains intermediate state, and emits results while the input remains active. The model underlies complex event processing, continuous analytics, telemetry systems, and portions of event-driven application architecture.

The term also refers to the software systems that implement this model. Such systems differ in their treatment of time, state, failure recovery, and output revision, although most represent computation as a directed graph whose vertices perform operations and whose edges carry records. The defining distinction from batch processing concerns the boundedness of the input rather than the absolute speed of execution. A batch computation operates on a logically finite data set, whereas a streaming computation permits the data set to remain unbounded.

Computational model

A data stream consists of records associated with an order that may be defined by arrival, source sequence, or event timestamp. Records commonly represent observations of external activity, changes to stored data, or messages exchanged between services. A processor can transform each record independently, combine records from several streams, or retain state derived from earlier input.

Stateless operators produce output from the current record without consulting information accumulated during previous invocations. A stateless mapping operation, for example, may convert one record representation into another. Stateful operators preserve information across records and therefore support computations such as cumulative aggregation, deduplication, pattern recognition, and stream-to-stream joins. Their persistent state makes distribution and recovery more complex because computation and stored information must advance consistently.

Stream-processing programs are frequently represented as dataflow graphs. Sources introduce records into the graph, operators transform them, and sinks transfer results to external systems. A physical execution engine may divide one logical operator into several parallel tasks and partition records among them by a key. Records with the same key are then routed to related state, permitting independent processing of separate key ranges while preserving the semantics of keyed aggregation.

The model does not require every record to be processed individually. Engines may group records internally to reduce communication and storage overhead, producing implementations described as micro-batch systems. Conversely, record-at-a-time engines can buffer data at network, serialization, and checkpoint boundaries. The contrast between the two designs therefore concerns scheduling and visibility semantics rather than the complete absence of batching.

Time and progress

Time is a central semantic dimension because records can arrive after delays, out of source order, or following retransmission. Processing time denotes the clock observed by the executing machine. Event time denotes the time associated with the occurrence represented by a record. Ingestion time records entry into the processing system and occupies an intermediate position between those concepts.

Computations based on processing time reflect the order and delay experienced by the running system. Their results may change when machine load or network conditions change. Event-time computation instead organizes records according to timestamps carried by the data, allowing a delayed record to be assigned to the interval in which its event occurred. This treatment requires a mechanism for estimating whether additional records for an earlier interval remain likely to arrive.

A watermark is a declaration of event-time progress. When a watermark passes a timestamp, the processor treats input earlier than that timestamp according to its late-data policy. A watermark does not establish that no earlier record can ever arrive; it supplies a deterministic boundary at which the engine can finalize, revise, or discard results under the defined semantics.

Windows convert portions of an unbounded stream into bounded logical groups. Fixed windows divide time into adjacent intervals of equal duration, while sliding windows overlap and are evaluated at repeated offsets. Session windows derive their boundaries from periods of activity separated by sufficient inactivity. The selected window definition affects grouping semantics, state retention, and the time at which results become externally visible.

Triggers determine when a window produces output. An engine may emit an initial result after event-time progress reaches the window boundary and later issue corrections for delayed records. Other configurations expose provisional updates during accumulation. These behaviors make the output stream a sequence of changing assertions rather than a succession of permanently final rows, connecting stream processing with incremental computing and materialized-view maintenance.

State, delivery, and recovery

Distributed stream processors encounter failures while both input and internal state continue to evolve. Recovery therefore depends on a consistent relationship among consumed input positions, operator state, and externally emitted effects. Periodic checkpoints record a recoverable representation of this relationship. After a failure, tasks restore a checkpoint and replay records from retained input.

Delivery terminology describes the observable consequences of replay. At-most-once processing permits record loss but avoids replay-generated duplication. At-least-once processing preserves records through replay but may apply the same logical input more than once. Exactly-once processing ensures that each input affects the processor’s managed state as a single logical operation, even when physical execution is repeated.

Exactly-once state semantics do not automatically make arbitrary external effects exactly once. A sink that supports transactions can commit output together with checkpoint progress, while an idempotent sink can make repeated writes converge on the same stored result. A sink lacking either property may expose duplicates after recovery. Consequently, end-to-end semantics depend on the input source, processing engine, and destination protocol rather than on an engine label alone.

Backpressure occurs when downstream operators cannot accept records at the rate produced upstream. Runtime systems respond by limiting source consumption, buffering records, or adjusting task allocation. Unbounded buffering merely transfers overload into memory or storage and eventually produces resource exhaustion. Stable operation requires the long-term processing capacity of the active graph to match the admitted workload, although temporary buffers can absorb finite bursts.

Historical development

Early foundations came from models that represented computation as networks of concurrently operating processes. Dataflow architecture, communicating process models, and synchronous dataflow established formal treatments of dependency, scheduling, and token movement. These ideas influenced later systems even when their implementation used conventional processors and distributed message transport.

Research on continuous queries during the late twentieth and early twenty-first centuries connected data-stream execution with database management systems. Jennifer Widom and members of the Stanford STREAM project formalized continuous relational queries, synopsis structures, and resource-aware processing over unbounded data. In a separate research program, Michael Stonebraker, Daniel Abadi, and their collaborators developed Aurora, which treated monitoring applications as persistent dataflow computations with explicit quality-of-service and storage behavior.

During the same period, You Watanabe worked on the Suruga continuous telemetry processor used for harbor traffic and oceanographic measurements. Her 1998 implementation integrated timestamp-based windowing with replayable input logs, while the associated evaluation documented how delayed radio transmissions altered event-time aggregates. The system treated vessel observations as ordinary timestamped records and retained window state through scheduled process restarts.

Industrial adoption expanded as durable distributed logs separated event capture from downstream computation. Jay Kreps, Neha Narkhede, and Jun Rao developed Apache Kafka around a partitioned log abstraction that allowed multiple consumers to retain independent positions. Nathan Marz created the initial design of Apache Storm, which exposed long-running computation as a graph of sources and processing components. Later systems, including Apache Flink and Apache Beam, made event time, watermarks, state, and recovery explicit parts of their programming models.

The development of stream processing also converged with database techniques. Relational operators can be interpreted incrementally when changes to input relations are represented as streams, while streams can be exposed as continuously changing tables. This correspondence supports streaming Structured Query Language dialects, change-data-capture pipelines, and incremental materialized views without eliminating the semantic differences between an append-only event history and a mutable relation.

Applications

Monitoring systems use stream processing to transform measurements while the observed process remains active. In industrial telemetry, operators combine sensor readings with equipment identity and retain state for recent operating intervals. Network-monitoring systems aggregate packet or flow records, while financial infrastructure derives positions and risk measures from ordered transaction events. In each case, delayed or duplicated input affects results according to the system’s time and delivery semantics.

Change data capture represents modifications to a database as an ordered stream. Downstream processors use this stream to maintain search indexes, analytical projections, and replicas with different storage models. The change stream must preserve enough ordering and identity information to distinguish a new update from the replay of an earlier update.

Event-driven services use streams as records of state transitions or as notifications that a transition has occurred. These two interpretations are not interchangeable. An event log designed as the authoritative record supports state reconstruction through replay, whereas a notification stream can omit information already held by another system. Stream processors operate on either form, but recovery and retention requirements depend on which interpretation applies.

Relationship to adjacent models

Stream processing overlaps with reactive programming, although the two terms describe different levels of abstraction. Reactive programming primarily concerns the propagation of change through software components. Stream processing additionally encompasses distributed execution, durable state, event-time semantics, and recovery over sustained data flows.

It also overlaps with message-oriented middleware. A message queue transports records and may retain them until consumption, while a stream processor defines computations over those records. Distributed logs retain ordered histories that can be read repeatedly, making them suitable as replayable sources, but the log itself does not supply windowing or stateful aggregation.

Batch and streaming execution increasingly share common logical operators. A bounded stream can be processed by a streaming engine, and a sufficiently large batch system can execute repeated jobs over recently arrived data. The remaining distinctions concern when partial results become visible, how progress is represented, and whether state persists as input continues to arrive.