I am designing a distributed system where a producer streams a finite batch of records across a multi-partition Apache Kafka topic. Downstream consumers in a single consumer group process these records. We need a reliable way for consumers to detect when the entire batch has finished processing across all partitions.

One proposed approach is to have the producer broadcast an identical "End of Batch" marker event to every single partition in the topic simultaneously, allowing individual consumer instances to detect the end-of-stream boundary locally.

I have concerns about this approach regarding consumer group rebalances: if a rebalance occurs mid-batch and a partition is reassigned after its marker has already been consumed and committed by a previous instance, the newly assigned consumer instance will never see the marker, causing the pipeline to hang.

Question: Does broadcasting duplicate end-of-stream markers across all partitions violate core Kafka consumer group mechanisms (such as rebalances), and is this documented as a known anti-pattern in distributed stream processing?