topic Re: Permanently add python file path to sys.path in Databricks in Data Engineering

Permanently add python file path to sys.path in Databricks

Direo — Tue, 21 Jun 2022 21:56:11 GMT

If your notebook is in different directory or subdirectory than python module, you cannot import it until you add it to the Python path.

That means that even though all users are using the same module, but since they are all working from different repos, they cannot import it until they add the path.

I wonder maybe it is possible to add module file path to Databricks sys.path permanently or until the file is deleted.

Re: Permanently add python file path to sys.path in Databricks

Hubert-Dudek — Wed, 22 Jun 2022 17:53:06 GMT

You can use it inside the same repo. Provide a whole path from the highest repo level in any notebook inside the repo. As you mentioned, if the file is in another repo, you need to use sys.path.append. To make it permanent, you can try to edit global init scripts.

from directory.sub_directory.my_file import MyClass
 
"""
Repo
-------\directory
------------------\sub_directory
-------------------------------------\my_file 
"""

Re: Permanently add python file path to sys.path in Databricks

Prabakar — Sat, 16 Jul 2022 00:15:55 GMT

@Direo Direo you can refer to this. The feature is now public preview.

Re: Permanently add python file path to sys.path in Databricks

uzadude — Tue, 13 Dec 2022 12:20:28 GMT

Hi, the init_script doesn't work for me (worker's pythonpath doesn't get affected).

and the suggested options in the above link don't help either.

is there a way to add another folder to the PYTHONPATH of the workers?

Re: Permanently add python file path to sys.path in Databricks

Cintendo — Sun, 25 Dec 2022 22:21:53 GMT

For worker node, you can set spark config in cluster setting: spark.executorEnv.PYTHONPATH

However you need to make sure you append your Workspace path at the end as worker node needs other system python path.

This seems to be a hack to me. I hope databricks can respond with a more solid solution.

Re: Permanently add python file path to sys.path in Databricks

uzadude — Mon, 26 Dec 2022 07:12:48 GMT

setting the `spark.executorEnv.PYTHONPATH` did not work for me. it looked like Spark/Databricks overwrite this somewhere. I used a simple python UDF to print some properties like `sys.path` and `os.environ` and didn't see the path I added.

Finally, I found a hacky way of using `spark._sc._python_includes`.

you can see my answer to my self here

Re: Permanently add python file path to sys.path in Databricks

Cintendo — Mon, 26 Dec 2022 13:55:27 GMT

Thanks @Ohad Raviv . I will try your approach.

spark.executorEnv.PYTHONPATH works only for worker node not driver node. And it needs to set at the cluster initialization stage (under Spark tab). After cluster initialized, databricks overwrite it even if you manually do spark.conf.set.

I prefer setting environment not thru code as codying it breaks the code integrity. It is hard to enforce it when multiple people working on the same cluster. I wish there is a better way in databricks cluster screen, it allows users to append sys.path after the default; or allow people to do editable install (pip install -e) during development.

I checked the worker node PYTHONPATH using the following to make sure it gets appended.

def getworkerenv():

import os

return(os.getenv('PYTHONPATH'))

sc = spark.sparkContext

sc.parallelize([1]).map(lambda x: getworkerenv()).collect()

Re: Permanently add python file path to sys.path in Databricks

uzadude — Mon, 26 Dec 2022 17:21:21 GMT

the hacky solution above is meant to be used only while developing my own python module - this way I can avoid packaging a whl, deploying to the cluster, restarting the cluster and even restarting the notebook interpreter.

I agree that it is not suited for production. For that I would use either a whl ref in the workflow file or just prepare a docker image.

Re: Permanently add python file path to sys.path in Databricks

Jfoxyyc — Thu, 29 Dec 2022 07:25:39 GMT

To be honest I'm just inspecting which repo folder I'm running from (dev/test/prod) and sys.path.appending an appropriate path before importing my packages. Seems to work and its covered by the Terraform provider.

Re: Permanently add python file path to sys.path in Databricks

uzadude — Thu, 29 Dec 2022 16:29:21 GMT

The issue with that is that the driver's sys.path is not added to the executors' sys.path, and you could get "module not found" error if your code tries to import one of your modules.

but it will work fine for simple code that is self-contained.

Re: Permanently add python file path to sys.path in Databricks

Jfoxyyc — Thu, 29 Dec 2022 21:14:33 GMT

I've been successfully using this in Delta Live Table pipelines with many nodes. Seems to work for my use case.