cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Asset Bundles + GitHub Actions: why does bundle deploy re-create UC schema and volume every run?

Ale_Armillotta
Contributor III

Hi everyone,
I’m working with Databricks Asset Bundles and deploying via GitHub Actions (CI/CD). I’m seeing behavior I don’t fully understand.

On every pipeline run (fresh git checkout/pull and then databricks bundle deploy to the same target environment), the deploy tries to create again my Unity Catalog resources — specifically a schema and a volume. I’d expect the deployment to be mostly idempotent and only apply deltas, but it looks like it re-applies the “create” actions each time.

A few questions:

  • Is this expected behavior for Asset Bundles (i.e., deploy always “ensures” resources exist, so it may attempt creation every run)?

  • Does the bundle keep any state or resource tracking across runs, or is it purely declarative each time (especially in CI like GitHub Actions)?

  • Are there recommended patterns to avoid re-creating / re-attempting creation of UC schema/volume on every deploy (e.g., separating infra from app deploy, using mode: production, structuring targets differently, etc.)?

If useful, I can share:

  • a simplified databricks.yml snippet (resources + targets),

  • the relevant GitHub Actions step,

  • and the exact deploy logs/error message.

The strange thing is that also the documentation on databricks says that the resoruces into databricks.yml won't destroy the ones already created.

 

 

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

stbjelcevic
Databricks Employee
Databricks Employee

Hi @Ale_Armillotta ,

To answer each of your questions:

  • Is this expected behavior for Asset Bundles
    • Yes, deploy is declarative and will attempt “create” whenever the bundle’s tracked state doesn’t already include that resource (names aren’t used to correlate).
       
  • Does the bundle keep any state or resource tracking across runs, or is it purely declarative each time
    • Yes, bundles persist state in the workspace and track resources by ID. If the bundle identity or state doesn’t match across runs, it treats resources as new.
       
  • Are there recommended patterns to avoid re-creating / re-attempting creation of UC schema/volume on every deploy
    • Use bind for preexisting resources, keep a stable bundle name/target/workspace and deployer, avoid dev prefixes in CI, preview with bundle plan, and optionally separate UC infra with lifecycle.prevent_destroy.

 

View solution in original post

2 REPLIES 2

Ale_Armillotta
Contributor III

I just tried to change the schema and the deploy is ok, because the DAB creates the Volume and schema. I tried a second time and also this time is ok. I'm not understanding why before it didn' work. Not it's just for my interest

stbjelcevic
Databricks Employee
Databricks Employee

Hi @Ale_Armillotta ,

To answer each of your questions:

  • Is this expected behavior for Asset Bundles
    • Yes, deploy is declarative and will attempt “create” whenever the bundle’s tracked state doesn’t already include that resource (names aren’t used to correlate).
       
  • Does the bundle keep any state or resource tracking across runs, or is it purely declarative each time
    • Yes, bundles persist state in the workspace and track resources by ID. If the bundle identity or state doesn’t match across runs, it treats resources as new.
       
  • Are there recommended patterns to avoid re-creating / re-attempting creation of UC schema/volume on every deploy
    • Use bind for preexisting resources, keep a stable bundle name/target/workspace and deployer, avoid dev prefixes in CI, preview with bundle plan, and optionally separate UC infra with lifecycle.prevent_destroy.