topic Re: Asset Bundles git branch per target in Get Started Discussions

Asset Bundles git branch per target

thibault — Fri, 08 Mar 2024 14:33:58 GMT

Hi,

I am migrating from dbx to Databricks Asset Bundles (DAB) a deployment setup where I have specific parameters per environment. This was working well with dbx, and I am trying now to define those parameters defining targets (3 targets : dev, uat, prod). Each of these targets should use a different git branch to pull the code from. However adding this :

git:
  branch: dev | uat | prod

It doesn't change anything, and no matter which target I deploy, the default branch specified in the main resource setup gets pulled :

git_source:
  git_url: *****
  git_provider: *****
  git_branch: main

Does anyone have a solution to pull from a different git branch based on which target is deployed via databricks bundle deploy -t target ?

Re: Asset Bundles git branch per target

thibault — Wed, 13 Mar 2024 13:09:43 GMT

Kaniz, thank you for your response.

I have done exactly what you suggest (both with naming the section git and git_source) and I am getting this error:

Error: terraform apply: exit status 1

Error: cannot update job: Invalid use of git source in Python Task specification

The workflow consists of 3 tasks : 2 python tasks and 1 notebook task, all pulled from git.

Re: Asset Bundles git branch per target

thibault — Wed, 13 Mar 2024 18:06:15 GMT

To add some details here, if I specify the git config under the overall resources: -> jobs, the deployment succeeds, but the git setups per branch do not override that global setup.

Also, if there is no git setup at the top level resources: -> jobs, despite it being defined at the target level, the bundle.tf.json produced with databricks bundle deploy does not contain any info about git sources, which is why the deployment fails.

Re: Asset Bundles git branch per target

HazelMiller — Thu, 21 Mar 2024 07:26:14 GMT

All good now?

Re: Asset Bundles git branch per target

thibault — Thu, 21 Mar 2024 09:14:30 GMT

Nope, I had to revert back to using dbx, so I will postpone migration to DAB until I find examples that work.

Re: Asset Bundles git branch per target

thibault — Thu, 28 Mar 2024 10:17:03 GMT

Something must have changed in the meantime on Databricks side. I have only updated databricks CLI to 016 and now, using a git / branch under each target deploys this setup, where feature-dab is the branch I want the job to pull sources from, I see this :

But not sure what this means as I've never seen this in the Jobs UI, and the YAML produced still shows only the main branch as source. Do you know @Retired_mod ? I can't find docs on this.

Re: Asset Bundles git branch per target

Fiabane — Fri, 30 Aug 2024 17:01:25 GMT

Hello Kaniz_Fatma,

I tried your suggestions. However, the tags: git_branch, git_url, and git_provider are not recognized in the bundle.yml (https://docs.databricks.com/en/dev-tools/bundles/settings.html#git)

They are valid in the resource.yml context, but they do not work as Thibault mentioned.

My bundle.yml

bundle: name: alerts include: - resources/*.yml targets: prod: git: origin_url: ***** # provider: AWS CodeCommit branch: master mode: production workspace: host: ******

putting the following tags inside the resource.yml had no effect

git_source: git_url: ********* git_provider: AWS CodeCommit git_branch: master

I saw that was released a way to deploy a resource setting as source a Git provider (https://community.databricks.com/t5/data-engineering/databricks-asset-bundle-dab-from-a-git-repo/td-p/49501). However, the video and files didn't clearly show how it was done. I would appreciate it if you could please clarify this part.

Best,