Here’s a clear and well-formatted StackOverflow question based on your latest error:

Title: com.mongodb.spark.sql.connector.exceptions.ConfigException: Missing configuration for: collection when reading MongoDB in PySpark

Body:

I am trying to read data from MongoDB using PySpark 4.0.1 in an Airflow DAG. My Spark code looks like this:

from pyspark.sql import SparkSession

uri = "mongodb://it:pass@127.0.0.1:27017/authSource=myit&directConnection=true"
database = "myit_db"
collection = "myCollection"

spark = SparkSession.builder \
    .appName("MongoSparkConnector") \
    .getOrCreate()

df = spark.read.format("mongodb") \
    .option("spark.mongodb.read.connection.uri", uri) \
    .option("spark.mongodb.read.database", database) \
    .option("spark.mongodb.read.collection", collection) \
    .load()

df.show()

When I run this, I get the following error:

com.mongodb.spark.sql.connector.exceptions.ConfigException: Missing configuration for: collection

Notes:

• I am including the following jars in my Spark environment:

mongo-spark-connector_2.13-10.5.0.jar
mongodb-driver-core-5.6.1.jar
bson-5.6.1.jar

• Spark version: 4.0.1 (Scala 2.13)
• I’ve tried specifying the collection explicitly, but the error persists.

Question:

Why am I getting Missing configuration for: collection when I have specified it in the .option("spark.mongodb.read.collection", collection)? Do I need to change the option names, or is there a mismatch between the MongoDB Spark Connector version and Spark 4.0.1?