cancel
Showing results for 
Search instead for 
Did you mean: 
Community Articles
Dive into a collaborative space where members like YOU can exchange knowledge, tips, and best practices. Join the conversation today and unlock a wealth of collective wisdom to enhance your experience and drive success.
cancel
Showing results for 
Search instead for 
Did you mean: 

Backing up Unity Catalog volumes without melting the driver

DanielMcauley
Databricks Employee
Databricks Employee

Tables are the easy part of a Unity Catalog backup, because you have DEEP CLONE. Volumes have no equivalent. When I started on a self-service backup tool this year, the first version backed volumes up the way I suspect most people do: list the files with dbutils.fs.ls and copy them one at a time with dbutils.fs.cp.

That’s fine for a few hundred files. On large volumes it ties up the driver and the run time keeps growing. Here’s what I changed.

What’s actually wrong with the loop

dbutils.fs.ls and dbutils.fs.cp only run on the driver. A serial loop means every byte goes through one machine, one file at a time, so you’re limited by per-file latency long before you use any real bandwidth. It also recopies everything every run, because it has no idea what’s already in the backup. The backup gets slower as the data grows, when it should be getting slower as the changes grow.

So the job is: copy only what changed, copy it in parallel, and where possible keep it off the cluster entirely.

Option 1: same dbutils, used differently (works on any cloud)

I kept this as the default because it needs nothing beyond Unity Catalog permissions. No storage keys, no service principal, and it runs the same on AWS, Azure or GCP.

List both the source and backup volumes once, building a map of relative path to (size, modification time). I walk the tree with an explicit stack rather than recursion, because some of these volumes are deeply nested. Then copy only files that are new, have a different size, or are newer than the backup copy, using a thread pool:

src=list_volume_files(src_root)   # {relpath: (size, mtime)}
dst = list_volume_files(dst_root)

todo = [rel for rel, (size, mtime) in src.items()
        if rel not in dst or dst[rel][0] != size or dst[rel][1] < mtime]

with ThreadPoolExecutor(max_workers=16) as pool:
    list(pool.map(lambda rel: dbutils.fs.cp(f"{src_root}/{rel}", f"{dst_root}/{rel}"), todo))

After the first full run, a nightly backup only moves the day’s changes. Permissions stay governed by READ VOLUME and WRITE VOLUME, which security teams like a lot more than storage keys in a notebook.

The catch is that bytes still go through the driver. Parallel copies fix latency, not bandwidth. For the really big volumes I needed something else.

Option 2: azcopy sync on Azure

azcopy sync copies directly between storage accounts. The data never touches the cluster, and it’s incremental out of the box.

I look up each volume’s real location from information_schema.volumes rather than hard-coding paths, convert the abfss://container@account.dfs.core.windows.net/... path to the https://account.blob.core.windows.net/container/... form, and run:

azcopy sync "<source>" "<backup>" --recursive --delete-destination=false

--delete-destination=false is on purpose, and I’d argue for it in any backup. If someone wipes a folder in the source, last night’s copy still needs to have it. Deletes should go through a separate retention process that someone actually approves. Run it with --dry-run first; it’s a quick way to see what would move.

Auth is a service principal with its secrets in a Databricks secret scope, or a managed identity.

The part that took me longest: installing azcopy

The clusters I was working with have no public internet access. Everything comes from an internal artifact repository. And they run Standard (shared) access mode. Two things bit me.

First, the init script has to pull the binary from the internal mirror. I put the download URL and an optional token behind environment variables, with Microsoft’s public URL as a fallback. The same script then works in both places.

Second, and this one isn’t well known: on Standard access mode, after init scripts finish, the platform re-locks root-created directories under /databricks and /tmp to 0700. A binary installed there looks fine from the init script, but the notebook user can’t execute it. /opt isn’t re-locked. I moved the install to /opt/azcopy, made it world-readable and executable, and the same script now works on both Dedicated and Standard clusters.

How I run it now

One parameter-driven notebook does tables (DEEP CLONE), volumes and SQL functions together, so a catalog gets one consistent backup run. Parallelism is capped at two levels, around 4 volumes and 16 files per volume. Without a cap, the bottleneck just moves to the driver or to storage throttling. Every setting is a widget (source catalog, schema filters, backup catalog, copy method), so other teams can use it without editing code. Since then I’ve packaged a generic version so it can be reused for other disaster-recovery setups.

There’s also a scheduled drift check comparing file counts and sizes between source and backup. I learned on the table side (see my post on gating deep clones) that a backup that looks fine and one that is fine aren’t always the same thing.

My rule of thumb: use the parallel dbutils copy everywhere, and switch the biggest Azure volumes to azcopy sync.

0 REPLIES 0