Asset Bundles git branch per target

[thibault]

Hi,

I am migrating from dbx to Databricks Asset Bundles (DAB) a deployment setup where I have specific parameters per environment. This was working well with dbx, and I am trying now to define those parameters defining targets (3 targets : dev, uat, prod). Each of these targets should use a different git branch to pull the code from. However adding this :

git:
  branch: dev | uat | prod

It doesn't change anything, and no matter which target I deploy, the default branch specified in the main resource setup gets pulled :

git_source:
  git_url: *****
  git_provider: *****
  git_branch: main

Does anyone have a solution to pull from a different git branch based on which target is deployed via databricks bundle deploy -t target ?

[Kaniz]

Hi @thibault, When working with Databricks Asset Bundles (DABs), you can indeed specify different git branches for different targets.

Let’s address this step by step:

1. Understanding Databricks Asset Bundles (DABs):

   • DABs are a powerful tool for managing complex data, analytics, and ML projects on the Databricks platform.
   • They streamline development by providing CI/CD capabilities through a concise YAML syntax.
   • Bundles include metadata alongside your project’s source files, making it easier to provision infras...¹.

2. Defining Git Branches for Different Targets:

   • To specify different git branches for each target (dev, uat, prod), you need to configure your DAB accordingly.
   • In your DAB configuration, you can override the default git branch specified in the main resource setup for each target.

3. Bundle Configuration YAML:

   • The bundle configuration YAML allows you to define settings for your DAB.

   • Here’s an example of how you can set up git branches per target:

     bundle:
       name: MyDAB
       # Other bundle settings...
     
     targets:
       dev:
         git:
           git_branch: dev
           git_provider: <provider>
           git_url: <git_url>
     
       uat:
         git:
           git_branch: uat
           git_provider: <provider>
           git_url: <git_url>
     
       prod:
         git:
           git_branch: prod
           git_provider: <provider>
           git_url: <git_url>
     

   • Replace <provider> and <git_url> with your actual Git provider and repository URL.

4. Explanation:

   • In the above configuration, each target (dev, uat, prod) specifies its own git branch.
   • When deploying to a specific target (e.g., databricks bundle deploy -t dev), Databricks will use the corresponding git branch defined for that target.
   • The git_source section in your main resource setup will be overridden by the target-specific configuration.

5. Deploying with Different Git Branches:

   • Now, when you deploy using databricks bundle deploy -t target, Databricks will pull the code from the specified git branch associated with that target.

Remember to replace placeholders (<provider> and <git_url>) with your actual Git provider details and repository URL. With this setup, you’ll be able to manage different git branches per environment using DABs²³. 

[thibault]

Kaniz, thank you for your response.

I have done exactly what you suggest (both with naming the section git and git_source) and I am getting this error:

Error: terraform apply: exit status 1

Error: cannot update job: Invalid use of git source in Python Task specification

The workflow consists of 3 tasks : 2 python tasks and 1 notebook task, all pulled from git.

[thibault]

To add some details here, if I specify the git config under the overall resources: -> jobs, the deployment succeeds, but the git setups per branch do not override that global setup.

Also, if there is no git setup at the top level resources: -> jobs, despite it being defined at the target level, the bundle.tf.json produced with databricks bundle deploy does not contain any info about git sources, which is why the deployment fails.

[HazelMiller]

All good now?

[thibault]

Nope, I had to revert back to using dbx, so I will postpone migration to DAB until I find examples that work.

[thibault]

Something must have changed in the meantime on Databricks side. I have only updated databricks CLI to 016 and now, using a git / branch under each target deploys this setup, where feature-dab is the branch I want the job to pull sources from, I see this :

But not sure what this means as I've never seen this in the Jobs UI, and the YAML produced still shows only the main branch as source. Do you know @Kaniz ? I can't find docs on this.