Databricks Community

uzadude · 12-13-2022

I'm trying to set PYTHONPATH env variable in the cluster configuration: `PYTHONPATH=/dbfs/user/blah`. But in the driver and executor envs it is probably getting overridden and i don't see it.`%sh echo $PYTHONPATH` outputs:`PYTHONPATH=/databricks/spar...

uzadude · 12-29-2022

The issue with that is that the driver's sys.path is not added to the executors' sys.path, and you could get "module not found" error if your code tries to import one of your modules.but it will work fine for simple code that is self-contained.

uzadude · 12-26-2022

the hacky solution above is meant to be used only while developing my own python module - this way I can avoid packaging a whl, deploying to the cluster, restarting the cluster and even restarting the notebook interpreter.I agree that it is not suite...

uzadude · 12-25-2022

setting the `spark.executorEnv.PYTHONPATH` did not work for me. it looked like Spark/Databricks overwrite this somewhere. I used a simple python UDF to print some properties like `sys.path` and `os.environ` and didn't see the path I added.Finally, I ...

uzadude · 12-13-2022

Update:At last found a (hacky) solution!in the driver I can dynamically set the sys.path in the workers with:`spark._sc._python_includes.append("/dbfs/user/blah/")`combine that with, in the driver:```%load_ext autoreload%autoreload 2```and setting: `...

uzadude · 12-13-2022

do you have any suggestions as to what should I run in the init-script?setting an env variable there has no effect as it cannot change the main process env.how would I add a library to the python path?and even if I could, it would be hard-coded libra...

Databricks Community

User Stats

User Activity

Adding to PYTHONPATH in interactive Notebooks

Re: Permanently add python file path to sys.path in Databricks

Re: Permanently add python file path to sys.path in Databricks

Re: Permanently add python file path to sys.path in Databricks

Re: Adding to PYTHONPATH in interactive Notebooks

Re: Adding to PYTHONPATH in interactive Notebooks