Batch processing runs on a schedule (e.g., nightly Hadoop jobs). Stream processing operates continuously on unbounded data streams, instantly reacting to events as they occur.

Module 1: Stateful Stream Processing

Stateless streaming (like AWS Lambda reading from Kinesis) is simple. Stateful streaming is complex because the processor needs to remember past events to compute aggregates (e.g., 'sum of transactions in the last 5 minutes').

Challenges in Stateful Streaming

  • Windowing: Handling tumbling, sliding, and session windows over time.
  • Late Data: Dealing with events that arrive out-of-order due to network delays.
  • Watermarks: A mechanism to tell the system when it's safe to assume no more late data will arrive for a specific time window.

Module 2: Apache Flink vs. Spark Streaming

Spark Streaming originally used 'micro-batching' (processing small batches every few seconds). Apache Flink was built from the ground up as a true native stream processor, capable of sub-millisecond latencies and exactly-once processing guarantees.

Flink_SQL.sqlsql
SELECT 
    user_id,
    TUMBLE_END(event_time, INTERVAL '1' HOUR) AS window_end,
    COUNT(*) AS click_count
FROM clicks
GROUP BY 
    TUMBLE(event_time, INTERVAL '1' HOUR),
    user_id;

Module 3: Real-Time OLAP

Instead of pushing streams directly to dashboards, modern architectures sink streaming data into real-time OLAP databases (like Apache Pinot or Apache Druid), which serve sub-second analytical queries directly to user-facing applications.