topic Re: Adding to PYTHONPATH in interactive Notebooks in Data Engineering

Adding to PYTHONPATH in interactive Notebooks

uzadude — Tue, 13 Dec 2022 11:11:20 GMT

I'm trying to set PYTHONPATH env variable in the cluster configuration: `PYTHONPATH=/dbfs/user/blah`. But in the driver and executor envs it is probably getting overridden and i don't see it.

`%sh echo $PYTHONPATH` outputs:

`PYTHONPATH=/databricks/spark/python:/databricks/spark/python/lib/py4j-0.10.9.5-src.zip:/databricks/jars/spark--driver--driver-spark_3.3_2.12_deploy.jar:/WSFS_NOTEBOOK_DIR:/databricks/spark/python:/databricks/python_shell`

and `import sys; print(sys.path)`:

```

'/databricks/python_shell/scripts', '/local_disk0/spark-c87ff3f0-1b67-4ec4-9054-079bba1860a1/userFiles-ea2f1344-51c6-4363-9112-a0dcdff663d0', '/databricks/spark/python', '/databricks/spark/python/lib/py4j-0.10.9.5-src.zip', '/databricks/jars/spark--driver--driver-spark_3.3_2.12_deploy.jar', '/databricks/python_shell', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '', '/local_disk0/.ephemeral_nfs/envs/pythonEnv-267a0576-e6bd-4505-b257-37a4560e4756/lib/python3.9/site-packages', '/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages', '/databricks/python/lib/python3.9/site-packages', '/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/databricks/python/lib/python3.9/site-packages/IPython/extensions', '/root/.ipython'

```

if i work from Repos it does add the repo to everywhere `/Workspace/Repos/user@domain.com/my_repo`, but then i need all my modules to be straight there and it is not convenient.

please let me know if there's a work-around to set a `/dbfs/` path in all nodes without ugly trick of ***** UDF, but straight from the cluster init script or the best would be dynamic `spark.conf` property.

Re: Adding to PYTHONPATH in interactive Notebooks

Harun — Tue, 13 Dec 2022 11:17:30 GMT

Hi @Ohad Raviv can you try init-scripts, it might help you. https://docs.databricks.com/clusters/init-scripts.html

Re: Adding to PYTHONPATH in interactive Notebooks

uzadude — Tue, 13 Dec 2022 16:38:53 GMT

do you have any suggestions as to what should I run in the init-script?

setting an env variable there has no effect as it cannot change the main process env.

how would I add a library to the python path?

and even if I could, it would be hard-coded library and I would then need a dedicated cluster configuration for every developer/library.

Re: Adding to PYTHONPATH in interactive Notebooks

uzadude — Wed, 14 Dec 2022 07:50:44 GMT

Update:

At last found a (hacky) solution!

in the driver I can dynamically set the sys.path in the workers with:

`spark._sc._python_includes.append("/dbfs/user/blah/")`

combine that with, in the driver:

```

%load_ext autoreload

%autoreload 2

```

and setting: `spark.conf("spark.python.worker.reuse", "false")`

and we have a fully interactive Spark session with the ability to change python module code without the need to restart the Spark Session/Cluster.

Re: Adding to PYTHONPATH in interactive Notebooks

Harun — Wed, 14 Dec 2022 12:19:06 GMT

Thats great, Thanks for sharing the solution.

Re: Adding to PYTHONPATH in interactive Notebooks

Cintendo — Mon, 26 Dec 2022 14:06:30 GMT

init script won't work if you meant export PYTHONPATH env setting. Databricks shell overwrites it when it starts the python interpreter. One approach we make it work is if the code is under /dbfs, we do editable install at init script, e.g.

pip install -e /dbfs/some_repos_code

this creates a easy-install.pth under /databricks/python3 site-packages at cluster initialization, which will append to sys.path to driver and worker.

This approach avoids appending sys.path everywhere in the code, which breaks the code integrity; easier to enforce at cluster level.

We also tried to do the same editable install for Repos under /Workspace but failed. Apparently /Workspace partition is not mounted during cluster initialization. We are going to request databricks to look into this.