cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to Handle Versioning in Databricks Asset Bundles?

saniok
New Contributor II

 

Hi everyone,

In our organization, we are transitioning from defining Databricks jobs using the UI to managing them with asset bundles. Since asset bundles can be deployed across multiple workspacesโ€”each potentially having multiple targets (e.g., staging, production)โ€”I am exploring the best way to handle versioning.

Even within the same workspace, staging and production targets may require different bundle versions. Is there a built-in way to manage versioning in Databricks asset bundles by default?

One potential workaround is to pass the version as a parameter and use it in the target.presets.name. However, this approach requires specifying the version during deployment, which adds an extra step.

Ideally, Iโ€™d like to have the version integrated directly into the bundle itself so that itโ€™s automatically managed and consistent across deployments.

Has anyone encountered and solved this challenge? Any suggestions or best practices would be greatly appreciated!

Thanks in advance!

 

2 REPLIES 2

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @saniok,

 

  • In databricks.yml file you can include version information in this file to manage different versions of your bundles.
  • Example:
    bundle:
      name: my-bundle
      version: 1.0.0
      resources:
        jobs:
          my-job:
            name: my-job
            tasks:
              - task_key: my-task
                existing_cluster_id: 1234-567890-abcde123
                notebook_task:
                  notebook_path: ./my_notebook.py
     

 

Hi @Alberto_Umana ,

 

Thank you for the response.

Following your suggestion i'm getting a warning for validate command:

> databricks bundle validate

Warning: unknown field: version
at bundle
in databricks.yml:3:3

Name: my-bundle
Target: default

I'm using the latest version of databricks cli

> databricks -v
Databricks CLI v0.239.1

Do I miss something ? 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group