topic Re: How to Handle Versioning in Databricks Asset Bundles? in Data Engineering

How to Handle Versioning in Databricks Asset Bundles?

saniok — Fri, 24 Jan 2025 15:06:19 GMT

Hi everyone,

In our organization, we are transitioning from defining Databricks jobs using the UI to managing them with asset bundles. Since asset bundles can be deployed across multiple workspaces—each potentially having multiple targets (e.g., staging, production)—I am exploring the best way to handle versioning.

Even within the same workspace, staging and production targets may require different bundle versions. Is there a built-in way to manage versioning in Databricks asset bundles by default?

One potential workaround is to pass the version as a parameter and use it in the target.presets.name. However, this approach requires specifying the version during deployment, which adds an extra step.

Ideally, I’d like to have the version integrated directly into the bundle itself so that it’s automatically managed and consistent across deployments.

Has anyone encountered and solved this challenge? Any suggestions or best practices would be greatly appreciated!

Thanks in advance!

Re: How to Handle Versioning in Databricks Asset Bundles?

Alberto_Umana — Fri, 24 Jan 2025 15:27:24 GMT

Hi @saniok,

In databricks.yml file you can include version information in this file to manage different versions of your bundles.

Example:

bundle:
  name: my-bundle
  version: 1.0.0
  resources:
    jobs:
      my-job:
        name: my-job
        tasks:
          - task_key: my-task
            existing_cluster_id: 1234-567890-abcde123
            notebook_task:
              notebook_path: ./my_notebook.py

Re: How to Handle Versioning in Databricks Asset Bundles?

saniok — Mon, 27 Jan 2025 11:11:58 GMT

Hi @Alberto_Umana ,

Thank you for the response.

Following your suggestion i'm getting a warning for validate command:

> databricks bundle validate

Warning: unknown field: version
at bundle
in databricks.yml:3:3

Name: my-bundle
Target: default

I'm using the latest version of databricks cli

> databricks -v
Databricks CLI v0.239.1

Do I miss something ?