Databricks Asset Bundles: issue with python_file path in spark_python_task

[ext07_rvoort]

Hi,

I am trying to run a python file which is stored in src folder. However, I am getting the following error: Error: cannot update job: Invalid python file reference: src/get_git_credentials.py. Please visit the Databricks user guide for supported python references

Here is my task definition:

resources:
  jobs:
    clicksend_elt:
      name: clicksend_elt
      schedule:
        quartz_cron_expression: 0 0 9-18 ? * MON-FRI *
        timezone_id: Europe/Luxembourg
        pause_status: ${var.clicksend_schedule_status}
      tasks:
        - task_key: get_git_credentials
          job_cluster_key: clicksend_job_cluster
          email_notifications: {}
          run_if: ALL_SUCCESS
          spark_python_task:
            python_file: src/get_git_credentials.py

And the absolute path of the file I am trying to run is (my DAB project being data_lakehouse):

In the databricks.yml file I include resources with:

include:
  - resources/*.yml

How should I input the path of the file? Somehow the only was that it works is by providing the absolute path:

python_file: "${workspace.root_path}/files/src/get_git_credentials.py"

[szymon_dybczak]

Hi @ext07_rvoort ,

Could you try specify it in relative way:

python_file: ../src/get_git_credentials.py

So, in my setup I have following directory structure:

root folder:
  - databricks.yml file
  - src/
  - resources/job.yml

And then to refer in job.yml to source files that reside in src in need to provide path in following way:

../src/ingestion/extraction.py