topic Databricks Asset Bundles in Data Engineering

Databricks Asset Bundles

Skr7 — Mon, 06 May 2024 10:00:39 GMT

Hi, I'm implementing Databricks Asset bundles, my scripts are in GitHub and my /resource has all the .yml of my Databricks workflow which are pointing to the main branch

git_source: git_url: https://github.com/xxxx git_provider: gitHubEnterprise git_branch: main

Now when I work with DABs, I create a feature branch and when I deploy and run my bundle, I want my databricks workflows to point to my feature branch, is there a way to dynamically change my git_branch to my feature branch when I deploy my bundle and run it.
Also, can we create a script to /resource/xx.yml mapping, the reason is that I don't want all my jobs to run when I run my bundle for a script change that is part of a single job
@Retired_mod Appreciate your input here

Re: Databricks Asset Bundles

JacekLaskowski — Wed, 17 Jul 2024 14:52:13 GMT

Why not use Substitutions and Custom variables that can be specified on command line using --var="<key>=<value>"?

With all the features your databricks.yml would look as follows:

variables:
  git_branch:
    default: main

git_source:
  git_url: https://github.com/xxxx
  git_provider: gitHubEnterprise
  git_branch: ${var.git_branch}

On command line you'd databricks deploy as follows:

databricks bundle deploy \
  --var="git_branch=features/my_great_feature"

Think of Databricks Asset Bundles as a tool to prepopulate job/workflow definition templates (based on databricks.yml, command line, env vars) so once the assets are deployed, they are going to be executed / run with the values (unless overridden by asset-specific means, like Run now with different parameters in Jobs).