I set up a workflow using 2 tasks. Just for demo purposes, I'm using an interactive cluster for running the workflow.

            {
                "task_key": "prepare",
                "spark_python_task": {
                    "python_file": "file:/Workspace/Repos/devops/mlhub-mlops-dev/src/src/prepare_train.py",
                    "parameters": [
                        "/dbfs/raw",
                        "/dbfs/train",
                        "/dbfs/train"
                    ]
                },
                "existing_cluster_id": "XXXX-XXXXXX-XXXXXXXXX",
                "timeout_seconds": 0,
                "email_notifications": {}
            }

As stated in the documentation, I set up the environment variable in the cluster ... this is the excerpt of the json definition of the cluster:

  "spark_env_vars": {
    "PYSPARK_PYTHON": "/databricks/python3/bin/python3",
    "PYTHONPATH": "/Workspace/Repos/devops/mlhub-mlops-dev/src"
  }

Then, when I execute the task of type Python, and I logged the contents of the sys.path I can't find the path configured in the cluster. If I log the contents of os.getenv('PYTHONPATH'), I get nothing. It looks like the environment variables set up at cluster level are not being promoted to the python task