I'm trying to understand the architecture of a stream processing framework (conceptually, not JDK implementation details).
My current understanding is:
A stream pipeline is a chain of stages like
filter() -> map() -> sorted() -> filter() -> collect().Intermediate operations are lazy.
During execution, each stage behaves like a sink that receives elements and forwards them downstream.
In parallel execution, the source is split into partitions, and each worker processes one partition through an identical pipeline.
For stateless operations like filter() and map(), this makes sense because each worker can process its partition independently.
My confusion starts with a stateful operation like sorted().
Suppose the pipeline is:
stream
.filter(...)
.map(...)
.sorted()
.filter(...)
.collect(...);
Each worker would produce a locally sorted result:
Worker 1: [2, 5, 8]
Worker 2: [1, 6]
Worker 3: [3, 4, 7]
Clearly these cannot simply be concatenated.
My questions are:
Who is responsible for merging these locally sorted results into a globally sorted stream?
How does the downstream
filter()receive a globally sorted stream when it has no knowledge thatsorted()existed upstream?Is there a conceptual "coordinator" that introduces a synchronization point after stateful operations, performs the merge, and only then resumes execution of downstream stages?
More generally, how does a stream processing framework decide where merging is required versus where partial results can simply be concatenated?
I'm not looking for JDK-specific classes or implementation details. I'm trying to understand the general architecture and execution model that a stream framework would use to support pipelines containing both stateless and stateful operations.