Job fails on clusters only with library dependency

matmad
New Contributor III

Hello!

I have following problem: All my job runs fail when the job uses a library. Even the most basic job (print a string) and the most basic library package (no secondary dependencies, the script does not even import/use the library) fails with `Failed to reach the driver`:

matmad_0-1754491920584.png

* All my libraries are python wheels
* I use `spark_python_task` (but also tested `python_wheel_task` with same error)
* If I use serverless (same script, same .whl), all works fine
* If I remove the package from the job's library section, all works fine (as said: I don't even import/use the library)
* I also used a different python wheel package and created a wheel using https://docs.databricks.com/aws/en/jobs/how-to/use-python-wheels-in-workflows#step-6-run-the-job-and...
* It doesn't matter if I configure the job via yml in asset bundle or "manually" in the UI

The logs of the clusters don't really help me.

I really appreciate your ideas - thank you!

The script:

matmad_1-1754492300025.png

The job yml:

matmad_2-1754492381339.png