Hi. In Databricks workflows, I submit a spark job (Type = "Spark Submit"), and a bunch of parameters, starting with --py-files.

This works where all the files are in the same s3 path, but I get errors when I put a "common" module in a different s3 path:

"--py-files",
"s3://some_path/appl_src.py",
"s3://some_path/main.py",
"s3://a_different_path/common.py",

I get an error saying "common" doesn't exist, when I know in fact the path exists. From Standard output:

Traceback (most recent call last):
File "/local_disk0/tmp/spark-123/appl_src.py", line 21, in <module>
from common import my_functions
ModuleNotFoundError: No module named 'common'

Additionally log4j mentions the first two, but not the third:

24/02/28 21:41:00 INFO Utils: Fetching s3://some_path/appl_src.py to ...
24/02/28 21:41:00 INFO Utils: Fetching s3://some_path/main.py to ...

Why does Spark ignore the third argument? Or it has to be in the same s3 path?