I have set up a small Xubuntu machine with intention of making it my single-node playaround Spark cluster. The cluster seems to be set up correctly - I can access the WebUI at port 8080, it shows a single worker attached, and I can also telnet the IP address on the port 7077.

Then I set up a python venv, install some dependencies, like pyspark and py4j and execute following code:

from pyspark.sql import SparkSession
import findspark

if __name__ == "__main__":

    findspark.init()
    spark = SparkSession.builder.appName("test2").master("spark://xubuntu-mini:7077").getOrCreate()

And this leads to a following error:

25/12/25 11:10:10 WARN Utils: Your hostname, xubuntu-mini resolves to a loopback address: 127.0.1.1; using 192.168.0.81 instead (on interface enp3s0f0)
25/12/25 11:10:10 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/12/25 11:10:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Traceback (most recent call last):
  File "/home/sopel/./sparktest.py", line 7, in 
    spark = SparkSession.builder.appName("test2").master("spark://xubuntu-mini:7077").getOrCreate()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sopel/pyspark-test/lib/python3.12/site-packages/pyspark/sql/session.py", line 560, in getOrCreate
    session = SparkSession(sc, options=self._options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sopel/pyspark-test/lib/python3.12/site-packages/pyspark/sql/session.py", line 636, in __init__
    jSparkSessionClass.getDefaultSession().isDefined()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'JavaPackage' object is not callable

What is interesting, is that in SparkUI I can see that there is connectivity between my Python script and the Spark driver, because after the execution I can see this in the UI:

However, when I investingate deeper, the state of the executor for this application is "KILLED", its stdout log is empty, and its stderr log looks like following:

Spark Executor Command: "/usr/lib/jvm/java-17-openjdk-amd64/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=42133" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.0.81:42133" "--executor-id" "0" "--hostname" "192.168.0.81" "--cores" "4" "--app-id" "app-20251225110102-0000" "--worker-url" "spark://Worker@192.168.0.81:43017" "--resourceProfileId" "0"

I have a feeling that this is all not a huge issue, maybe some version incompatiblity or me doing something wrong. However at this point I have no more ideas.

What have I tried:

• Downgrading the JAva from 21 to 17 (apparently 21 is not compatibile with Spark 3.5.7)

• Connecting from the remote machine

Actually at this point I have no idea what can I try to resolve the issue. Thanks in advance for all your answers.

Version info:

Spark 3.5.7
openjdk 17.0.17 2025-10-21
OpenJDK Runtime Environment (build 17.0.17+10-Ubuntu-124.04)
OpenJDK 64-Bit Server VM (build 17.0.17+10-Ubuntu-124.04, mixed mode, sharing)
Ubuntu 24.04.3 LTS