Why planInputPartitions is called multiple times in a micro-batch?
06:46 12 Jul 2021

I'm developing a new Spark connector using data source v2 API (spark 3.1.1). I noticed that the planInputPartitions method (in MicroBatchStream) is called twice every micro-batch.

What the motivation/reason is?

apache-spark datasource