Databricks Asset Bundles: issue with python_file path in spark_python_task

[ext07_rvoort]

Hi,

I am trying to run a python file which is stored in src folder. However, I am getting the following error: Error: cannot update job: Invalid python file reference: src/get_git_credentials.py. Please visit the Databricks user guide for supported python references

Here is my task definition:

resources:
  jobs:
    clicksend_elt:
      name: clicksend_elt
      schedule:
        quartz_cron_expression: 0 0 9-18 ? * MON-FRI *
        timezone_id: Europe/Luxembourg
        pause_status: ${var.clicksend_schedule_status}
      tasks:
        - task_key: get_git_credentials
          job_cluster_key: clicksend_job_cluster
          email_notifications: {}
          run_if: ALL_SUCCESS
          spark_python_task:
            python_file: src/get_git_credentials.py

And the absolute path of the file I am trying to run is (my DAB project being data_lakehouse):

In the databricks.yml file I include resources with:

include:
  - resources/*.yml

How should I input the path of the file? Somehow the only was that it works is by providing the absolute path:

python_file: "${workspace.root_path}/files/src/get_git_credentials.py"

[szymon_dybczak]

Hi @ext07_rvoort ,

Could you try specify it in relative way:

python_file: ../src/get_git_credentials.py

So, in my setup I have following directory structure:

root folder:
  - databricks.yml file
  - src/
  - resources/job.yml

And then to refer in job.yml to source files that reside in src in need to provide path in following way:

../src/ingestion/extraction.py

[ext07_rvoort · Accepted solution]

Hi @szymon_dybczak,

Thanks for you reply. The issue is related to the fact if am also providing git_source parameters in this job (even though the source parameter of my task is set to WORKSPACE). When I comment the git_source part, specifying the path in a relative way works: python_file: ./../src/get_git_credentials.py

On this page I found this: "The job git_source field and task source field set to GIT are not recommended for bundles, because local relative paths may not point to the same content in the Git repository. Bundles expect that a deployed job has the same files as the local copy from where it was deployed." 

Instead, clone the repository locally and set up your bundle project within this repository, so that the source for tasks are the workspace."

I would like to configure this git_source parameters only for the task that requires it but it seems like it can only be defined at job level.