Databricks Secrets

[lessalucas]

Recently Databricks Secrets became generally available.

Now, you can manage your secrets whitin the catalog using the namespace: catalog.schema.secret

However, you can do the opposite of what you used to do: Connect Unity Catalog secrets to Azure Key Vault (and others). This way, you keep the values in KV while using UC to manage and reference these secrets .

Which one should I choose?

There's no single answer. Everything depends on how the management works in the company.

No one Data Engineer has permission to create key vaults and the SOS can take several days to be closed? Go with Databricks Secrets.

Do you need to centralize every kind of key in the company? Go with key vaults.

It all depends on the scope and context. You don't need to choose one OR the other. You can work both together too. 

[DoTA]

One dimension I would add, since it usually decides it for regulated shops: governance and audit.

- UC-managed secrets (catalog.schema.secret) inherit UC's permission model and land in the same audit / system tables as everything else, so "who read this secret and when" is answerable in one place. Workspace-scoped secret ACLs are coarser and sit outside UC.

- Azure Key Vault-backed scopes are effectively all-or-nothing: anyone with READ on the scope can read every secret in the backing vault, and the scope authenticates as a single identity. If you need per-secret separation you end up creating many small vaults, which gets unwieldy fast.

- KV-backed does give you one real advantage: rotation happens in Key Vault and consumers pick up the new value with no change on the Databricks side. With native secrets you have to push the new version yourself.

- At high job concurrency, watch KV throttling (429s). A few thousand concurrent tasks each fetching a secret at startup can hit the vault's rate limits; native secrets do not have that failure mode.

Where we landed: native UC secrets for things the data platform team creates and rotates itself (service tokens, connection strings), KV-backed for anything owned centrally by security that already had a rotation process. Agree it is not one OR the other.