Databricks Asset Bundles

Skr7
New Contributor II

‎05-06-2024 03:00 AM

Hi, I'm implementing Databricks Asset bundles, my scripts are in GitHub and my /resource has all the .yml of my Databricks workflow which are pointing to the main branch 

 

 

 

 

 git_source:
        git_url: https://github.com/xxxx
        git_provider: gitHubEnterprise
        git_branch: main

 

 

 

 

Now when I work with DABs, I create a feature branch and when I deploy and run my bundle, I want my databricks workflows to point to my feature branch, is there a way to dynamically change my git_branch to my feature branch when I deploy my bundle and run it.
Also, can we create a script to /resource/xx.yml mapping, the reason is that I don't want all my jobs to run when I run my bundle for a script change that is part of a single job 
@Retired_mod Appreciate your input here

0 Kudos

JacekLaskowski
Databricks MVP

‎07-17-2024 07:52 AM

Why not use Substitutions and Custom variables that can be specified on command line using --var="<key>=<value>"?

With all the features your databricks.yml would look as follows:

variables:
  git_branch:
    default: main

git_source:
  git_url: https://github.com/xxxx
  git_provider: gitHubEnterprise
  git_branch: ${var.git_branch}

On command line you'd databricks deploy as follows:

databricks bundle deploy \
  --var="git_branch=features/my_great_feature"

Think of Databricks Asset Bundles as a tool to prepopulate job/workflow definition templates (based on databricks.yml, command line, env vars) so once the assets are deployed, they are going to be executed / run with the values (unless overridden by asset-specific means, like Run now with different parameters in Jobs).

 

0 Kudos