Thanks @Ohad Raviv​ . I will try your approach.

spark.executorEnv.PYTHONPATH works only for worker node not driver node. And it needs to set at the cluster initialization stage (under Spark tab). After cluster initialized, databricks overwrite it even if you manually do spark.conf.set.

I prefer setting environment not thru code as codying it breaks the code integrity. It is hard to enforce it when multiple people working on the same cluster. I wish there is a better way in databricks cluster screen, it allows users to append sys.path after the default; or allow people to do editable install (pip install -e) during development.

I checked the worker node PYTHONPATH using the following to make sure it gets appended.

def getworkerenv():

  import os

  return(os.getenv('PYTHONPATH'))

sc = spark.sparkContext

sc.parallelize([1]).map(lambda x: getworkerenv()).collect()