cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

set PYTHONPATH when executing workflows

FranPérez
New Contributor III

I set up a workflow using 2 tasks. Just for demo purposes, I'm using an interactive cluster for running the workflow.

            {
                "task_key": "prepare",
                "spark_python_task": {
                    "python_file": "file:/Workspace/Repos/devops/mlhub-mlops-dev/src/src/prepare_train.py",
                    "parameters": [
                        "/dbfs/raw",
                        "/dbfs/train",
                        "/dbfs/train"
                    ]
                },
                "existing_cluster_id": "XXXX-XXXXXX-XXXXXXXXX",
                "timeout_seconds": 0,
                "email_notifications": {}
            }

As stated in the documentation, I set up the environment variable in the cluster ... this is the excerpt of the json definition of the cluster:

  "spark_env_vars": {
    "PYSPARK_PYTHON": "/databricks/python3/bin/python3",
    "PYTHONPATH": "/Workspace/Repos/devops/mlhub-mlops-dev/src"
  }

Then, when I execute the task of type Python, and I logged the contents of the sys.path I can't find the path configured in the cluster. If I log the contents of os.getenv('PYTHONPATH'), I get nothing. It looks like the environment variables set up at cluster level are not being promoted to the python task

7 REPLIES 7

tomasz
New Contributor III
New Contributor III

What documentation are you following here?

You shouldn't need to specify PYTHONPATH or PYSPARK_PYTHON as this section is for Spark specific environment variables such as "SPARK_WORKER_MEMORY".

FranPérez
New Contributor III

I'm following the standard Python documentation .. Databricks is compatible with Python AFAIK

This approach works when using "traditional" jobs, but not when using tasks in workflows

User16764241763
Honored Contributor

Could you please try this instead?

import sys

sys.path.append("/Workspace/Repos/devops/mlhub-mlops-dev/src")

You need to do sys.path.append in the udf if the lib need to available on workers.

from pyspark.sql.functions import *

def move_libs_to_executors():

import sys

sys.path.append("/Workspace/Repos/devops/mlhub-mlops-dev/src")

lib_udf = udf(move_libs_to_executors)

df = spark.range(100)

df.withColumn("lib", lib_udf()).show()

I'm already using this "fix", but this goes against good development practices because you are hardcoding a filepath in your code. This filepath should be provided via a parameter, this is the reason that in most solutions ENVIRONMENT VARIABLES are used for , because the path might change at deployment time.

And as I mentioned before, following the Databricks documentation, you should be able to set environment variables using the spark_env_vars section. Is there anything wrong with my initial approach?

tomasz
New Contributor III
New Contributor III

@Fran Pérez​ I did a little research on this and found that currently PYTHONPATH will be overwritten on cluster startup time and there is no way to redefine it at this time. At this point we would recommend using the already defined PYTHONPATH directories for your libraries or just using user libraries for this.

To see the PYTHONPATH that's set by default you can run:

%sh echo $PYTHONPATH

as a separate cell in a notebook that's attached to your cluster.

Cintendo
New Contributor III

This won't work for editable library as editable library is append path using site package from easy-install.pth

jose_gonzalez
Moderator
Moderator

Hi @Fran Pérez​,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!