Job deploy with git source using Asset Bundles

jonhieb — Fri, 21 Mar 2025 12:42:31 GMT

Hi, I'm trying to deploy a job with a notebook task based on a git source. But I'm facing an error when I try to deploy.

This is the YAML file:


resources:
jobs:
data_quality_pipelines_job:
name: schedule_data_quality_job

schedule:
quartz_cron_expression: "0 0 5 ? * Mon-Fri" # At 5:00:00am, every day between Monday and Friday, every month
timezone_id: "America/Sao_Paulo"

timeout_seconds: 7200 # 2 hours

git_source:
git_branch: main
git_provider: gitHub
git_url: https://github.com/xxxx/yyyyyyy

email_notifications:
on_failure:
- ${workspace.current_user.userName}
webhook_notifications:
on_failure:
- id: xxxxxx-xxxxxxx-xxxxxx

tasks:
- task_key: data_quality_task
pipeline_task:
pipeline_id: xxxxxxx-xxxxxxx-xxxxxxx
full_refresh: false

- task_key: notify_business_areas
depends_on:
- task_key: data_quality_task
notebook_task:
notebook_path: src/notifications/send_notifications.ipynb
max_retries: 0

parameters:
- name: notification_conf_file
default: ${var.notification_conf_file}

When I try to run it, this error appears:


Error: terraform apply: exit status 1

Error: cannot update job: Invalid notebook_path: src/notifications/send_notifications.ipynb. Only absolute paths are currently supported. Paths must begin with '/'.

with databricks_job.kpis_analytics_job,
on bundle.tf.json line 79, in resource.databricks_job.kpis_analytics_job:
79: }

If I try to add a '/', the tool will search at my Databricks Workspace, but I want to consume the notebook based on git. I know that the path exists on my git URL.

What should I do?

Re: Job deploy with git source using Asset Bundles

jonhieb — Mon, 07 Apr 2025 14:48:10 GMT

I found the solution for this one:

When we’re running a job that needs to consume files from a Git repo, in addition to declaring the git_source clause, we also need to declare the source clause within the task configuration.

The image below demonstrates an example:

Notice that inside the notebook_task clause, I declared the source. That was enough to make it work.

topic Job deploy with git source using Asset Bundles in Data Engineering

Job deploy with git source using Asset Bundles

Re: Job deploy with git source using Asset Bundles