Hi @maartenvr , hi @Debayan ,
Are there any updates on this? Have you found a solution, or can the problem at least be narrowed down to specific DBR versions? I am on a cluster with 11.3 LTS and deploy my custom packaged code (named simply 'src') as Python wheel using Databricks Asset Bundles. Even though the package is successfully installed and can generally be used by the job, execution fails as soon as I want to parallelize anything using PySpark. The module 'src' is not found by the executors, and also none of its dependencies (e.g., 'xgboost'). 

~

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 5285.0 failed 4 times, most recent failure: Lost task 1.3 in stage 5285.0 (TID 13412) (10.22.37.185 executor 2): org.apache.spark.api.python.PythonException: 'ModuleNotFoundError: No module named 'src''. Full traceback below: Traceback (most recent call last): File "/databricks/spark/python/pyspark/worker.py", line 1018, in main func, profiler, deserializer, serializer = read_command(pickleSer, infile) File "/databricks/spark/python/pyspark/worker.py", line 92, in read_command command = serializer.loads(command.value) File "/databricks/spark/python/pyspark/serializers.py", line 540, in loads return cloudpickle.loads(obj, encoding=encoding) ModuleNotFoundError: No module named 'src'