topic Re: set PYTHONPATH when executing workflows in Data Engineering

set PYTHONPATH when executing workflows

FranPérez — Mon, 01 Aug 2022 07:37:10 GMT

I set up a workflow using 2 tasks. Just for demo purposes, I'm using an interactive cluster for running the workflow.

            {
                "task_key": "prepare",
                "spark_python_task": {
                    "python_file": "file:/Workspace/Repos/devops/mlhub-mlops-dev/src/src/prepare_train.py",
                    "parameters": [
                        "/dbfs/raw",
                        "/dbfs/train",
                        "/dbfs/train"
                    ]
                },
                "existing_cluster_id": "XXXX-XXXXXX-XXXXXXXXX",
                "timeout_seconds": 0,
                "email_notifications": {}
            }

As stated in the documentation, I set up the environment variable in the cluster ... this is the excerpt of the json definition of the cluster:

  "spark_env_vars": {
    "PYSPARK_PYTHON": "/databricks/python3/bin/python3",
    "PYTHONPATH": "/Workspace/Repos/devops/mlhub-mlops-dev/src"
  }

Then, when I execute the task of type Python, and I logged the contents of the sys.path I can't find the path configured in the cluster. If I log the contents of os.getenv('PYTHONPATH'), I get nothing. It looks like the environment variables set up at cluster level are not being promoted to the python task

Re: set PYTHONPATH when executing workflows

tomasz — Wed, 03 Aug 2022 19:56:09 GMT

What documentation are you following here?

You shouldn't need to specify PYTHONPATH or PYSPARK_PYTHON as this section is for Spark specific environment variables such as "SPARK_WORKER_MEMORY".

Re: set PYTHONPATH when executing workflows

FranPérez — Thu, 04 Aug 2022 05:39:08 GMT

I'm following the standard Python documentation .. Databricks is compatible with Python AFAIK

This approach works when using "traditional" jobs, but not when using tasks in workflows

Re: set PYTHONPATH when executing workflows

User16764241763 — Thu, 04 Aug 2022 05:48:11 GMT

Could you please try this instead?

import sys

sys.path.append("/Workspace/Repos/devops/mlhub-mlops-dev/src")

You need to do sys.path.append in the udf if the lib need to available on workers.

from pyspark.sql.functions import *

def move_libs_to_executors():

import sys

sys.path.append("/Workspace/Repos/devops/mlhub-mlops-dev/src")

lib_udf = udf(move_libs_to_executors)

df = spark.range(100)

df.withColumn("lib", lib_udf()).show()

Re: set PYTHONPATH when executing workflows

FranPérez — Thu, 04 Aug 2022 06:25:52 GMT

I'm already using this "fix", but this goes against good development practices because you are hardcoding a filepath in your code. This filepath should be provided via a parameter, this is the reason that in most solutions ENVIRONMENT VARIABLES are used for , because the path might change at deployment time.

And as I mentioned before, following the Databricks documentation, you should be able to set environment variables using the spark_env_vars section. Is there anything wrong with my initial approach?

Re: set PYTHONPATH when executing workflows

tomasz — Fri, 05 Aug 2022 16:25:33 GMT

@Fran Pérez I did a little research on this and found that currently PYTHONPATH will be overwritten on cluster startup time and there is no way to redefine it at this time. At this point we would recommend using the already defined PYTHONPATH directories for your libraries or just using user libraries for this.

To see the PYTHONPATH that's set by default you can run:

%sh echo $PYTHONPATH

as a separate cell in a notebook that's attached to your cluster.

Re: set PYTHONPATH when executing workflows

jose_gonzalez — Tue, 30 Aug 2022 17:07:06 GMT

Hi @Fran Pérez,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Re: set PYTHONPATH when executing workflows

Cintendo — Mon, 26 Dec 2022 01:52:09 GMT

This won't work for editable library as editable library is append path using site package from easy-install.pth

Re: set PYTHONPATH when executing workflows

newenglander — Fri, 23 May 2025 20:01:12 GMT

Hi @tomasz , I am not the original poster but as it is now 3 years later I wanted to ask: is it still the case that PYTHONPATH cannot be modified from an init script in a way that won't be overwritten?

Is there a solution for putting a Workspace directory on python's path aside from explicitly modifying `sys.path` in every executable notebook or python script?

Would `pip install -e /Workspace/<etc>` work in an init script?

Thank you

Re: set PYTHONPATH when executing workflows

kenmyers-8451 — Wed, 04 Mar 2026 21:56:40 GMT

Just checking in again if there is a way to do this in the last few years? As Fran mentioned, `sys.path.append("/Workspace/Repos/devops/mlhub-mlops-dev/src")` is not a great "fix" for the reasons already mentioned.

I've found that you can do `pip install .` but only on runtime 16.4 (and probably higher).

The other option is to set the working directory to the Repos/ location, but this is not ideal when working on a team because then you'd all have to be deploying to the same location and overwriting each others work.

I am surprised there seems to be no way to simply append to the pythonpath.