Databricks Asset Bundles: How to manage dependencies between volumes/files and cluster creation?

[geramkumar]

Problem Statement:

I have a Databricks Asset Bundles (DAB) project with the following requirement:

I need to create an all-purpose cluster whose configuration references:

• An init script stored at /Volumes/<catalog>/<schema>/<volume>/scripts/init_script.sh
• A requirements.txt file stored at /Volumes/<catalog>/<schema>/<volume>/libraries/requirements.txt

The init script itself refers to the requirements.txt file.

Therefore, the cluster can only be created successfully after the schema, volume, folders, and files have been created.

Required Dependency Sequence:

The desired deployment sequence is:

1. Create the schema.
2. Create the volume under the schema.
3. Create the required folders in the volume.
4. Copy the required files (init_script.sh, requirements.txt, etc.) into the volume.
5. Create the all-purpose cluster referencing those volume paths.

DAB supports the creation of the schema and volume, but I don't see a native DAB mechanism to:

• Create folders/files inside a Unity Catalog volume as part of the bundle deployment.
• Execute an arbitrary shell script as a deployment step.
• Define the required dependency ordering between these operations and the cluster resource.

Current Workaround:

To handle this dependency, I have created a wrapper shell script that performs the deployment in multiple stages.

Step 1 – Disable cluster creation

Rename:

clusters.yml → clusters.yml.disabled

This prevents the cluster from being processed during the initial bundle deployment.

Step 2 – Deploy the DAB

Run databricks bundle deploy to create the schema and volume.

Step 3 – Populate the volume

Run a shell script that:

• Creates the required folders in the volume.
• Copies init_script.sh, requirements.txt, and other required files into the appropriate volume paths.

Step 4 – Re-enable cluster creation

Rename:

clusters.yml.disabled → clusters.yml

Step 5 – Deploy the DAB again

Run databricks bundle deploy again so that the cluster is created after all of its dependencies are available.

This approach works, but it requires maintaining a custom wrapper script and effectively performing two separate bundle deployments.

Question:

What is the recommended/best-practice approach in Databricks Asset Bundles for handling this type of dependency?

Specifically, is there a supported way to ensure that:

Schema → Volume → Volume files → Cluster

are provisioned in the correct order within a DAB deployment?

If DAB does not currently support this dependency/workflow natively, what would be the recommended architecture or deployment pattern for managing files that must exist in a Unity Catalog volume before a cluster is created?

Any guidance on the preferred approach would be appreciated.

[SumeshKashyap]

Great question — DABs don't currently expose a first-class dependency graph for "populate this volume, then create this cluster", since volume file population isn't a bundle resource type the way clusters/jobs/volumes are.

Two patterns that avoid the deploy-twice workaround:

1. Move init scripts to a Workspace path, not a Volume. Bundle sync/include uploads your source files to /Workspace/... as part of bundle deploy, before Terraform applies your resources/*.yml. If your cluster's init_scripts and requirements.txt reference a bundle-relative Workspace path instead of /Volumes/..., the file is guaranteed to exist before the cluster resource is created — no extra ordering logic needed. This sidesteps the problem entirely for anything that doesn't strictly need to live in a UC Volume.

2. If the files genuinely need to be in a Volume (e.g. consumed by jobs outside the bundle), wire the two-phase approach into one CI step instead of manual runs: add a small "provisioning" job resource that copies files into the volume via the Files API, and trigger databricks bundle run <job-key> as a pre-deploy step ahead of the main bundle deploy in your pipeline. Terraform does respect implicit ordering when one resource references another via bundle variable interpolation, but that only orders resource creation, not arbitrary file copies into it — so the two-step reality still applies there today.

Worth double-checking the latest Asset Bundles resource reference in case a native "file" resource type has shipped since — this area moves fast.

[geramkumar]

Thank you Sumesh. Let me explore the second option and check the latest DAB documentation aswell. Keep up the good work