topic Re: Databricks Asset Bundles: issue with python_file path in spark_python_task in Data Engineering

Databricks Asset Bundles: issue with python_file path in spark_python_task

ext07_rvoort — Thu, 09 Oct 2025 08:18:46 GMT

Hi,

I am trying to run a python file which is stored in src folder. However, I am getting the following error: Error: cannot update job: Invalid python file reference: src/get_git_credentials.py. Please visit the Databricks user guide for supported python references

Here is my task definition:

resources: jobs: clicksend_elt: name: clicksend_elt schedule: quartz_cron_expression: 0 0 9-18 ? * MON-FRI * timezone_id: Europe/Luxembourg pause_status: ${var.clicksend_schedule_status} tasks: - task_key: get_git_credentials job_cluster_key: clicksend_job_cluster email_notifications: {} run_if: ALL_SUCCESS spark_python_task: python_file: src/get_git_credentials.py

And the absolute path of the file I am trying to run is (my DAB project being data_lakehouse):

In the databricks.yml file I include resources with:

include: - resources/*.yml

How should I input the path of the file? Somehow the only was that it works is by providing the absolute path:

python_file: "${workspace.root_path}/files/src/get_git_credentials.py"

Re: Databricks Asset Bundles: issue with python_file path in spark_python_task

szymon_dybczak — Thu, 09 Oct 2025 08:46:05 GMT

Hi @ext07_rvoort ,

Could you try specify it in relative way:

python_file: ../src/get_git_credentials.py

So, in my setup I have following directory structure:

root folder:
- databricks.yml file
- src/
- resources/job.yml

And then to refer in job.yml to source files that reside in src in need to provide path in following way:

../src/ingestion/extraction.py

Re: Databricks Asset Bundles: issue with python_file path in spark_python_task

ext07_rvoort — Tue, 14 Oct 2025 15:10:39 GMT

Hi @szymon_dybczak,

Thanks for you reply. The issue is related to the fact if am also providing git_source parameters in this job (even though the source parameter of my task is set to WORKSPACE). When I comment the git_source part, specifying the path in a relative way works: python_file: ./../src/get_git_credentials.py

On this page I found this: "The job git_source field and task source field set to GIT are not recommended for bundles, because local relative paths may not point to the same content in the Git repository. Bundles expect that a deployed job has the same files as the local copy from where it was deployed."

Instead, clone the repository locally and set up your bundle project within this repository, so that the source for tasks are the workspace."

I would like to configure this git_source parameters only for the task that requires it but it seems like it can only be defined at job level.