topic Re: Managing "Secret" Injection in DABs across Dev/Stage/Prod in Data Engineering

Managing "Secret" Injection in DABs across Dev/Stage/Prod

Khasim_1 — Sat, 19 Sep 2026 16:02:36 GMT

Hi everyone, I am fully moving our team to Databricks Asset Bundles (DABs) for CI/CD, but I’m struggling with the "Secret" management pattern.

How are you handling the injection of secrets (like API keys for ingestion) into DABs without hardcoding anything in the databricks.yml?
Do you use Databricks Secret Scopes referenced via environment variables in the bundle, or are you pulling them dynamically from an external vault (e.g., KeyVault/Secrets Manager) during the deploy phase?
What is the cleanest way to maintain "Environment Parity" for these secrets without having to manually configure scopes in each workspace?

Re: Managing "Secret" Injection in DABs across Dev/Stage/Prod

ThomazNeto — Sun, 20 Sep 2026 11:26:49 GMT

Hi Khasim,

No secret value ever goes into databricks.yml. Bundle variables get resolved at deploy time and rendered into the job definition in the workspace, so anyone who can open the job can read them. Only names and references live in the bundle.

Second, let the bundle own the secret containers, one per target. Two ways to do that today.

The classic route is secret scopes as a bundle resource (CLI 0.252 or newer):

resources:
secret_scopes:
ingestion:
name: ingestion-secrets
permissions:
- level: READ
group_name: data-engineers

On Azure we set backend_type: AZURE_KEYVAULT and keyvault_metadata (dns_name and resource_id) pointing to a different Key Vault per target, so values never touch the bundle or the CI runner. On AWS we keep Databricks-backed scopes and run databricks secrets put-secret right after bundle deploy, pulling the values from the pipeline's own secret store. Bundle owns the scope and ACLs, pipeline owns the values.

The newer route is Unity Catalog secrets as a bundle resource, added in CLI 1.12 last month:

resources:
secrets:
ingestion_api_key:
catalog_name: ${var.catalog}
schema_name: secrets
name: ingestion_api_key
value: ${var.ingestion_api_key}
grants:
- principal: data-engineers
privileges: [READ_SECRET]

The value has to be a variable reference. If you type a literal the CLI stops you with "Secret value must be a variable reference... Plain text secret values are not allowed". In CI you set BUNDLE_VAR_ingestion_api_key from your pipeline secrets and the value exists only there and in UC. Jobs read it with dbutils.secrets.get(catalog=..., schema=..., key=...). Two things to know before picking this: it needs DBR 17.3 LTS or serverless environment 4, and it doesn't work on SQL warehouses, init scripts or spark_env_vars. If you still need the double-brace secrets syntax in cluster env vars, keep those on scopes.

Third, jobs reference the scope or the catalog.schema, never the value. We pass the scope name as a job parameter with ${resources.secret_scopes.ingestion.name} and call dbutils.secrets.get at runtime.

Parity then takes care of itself. Same names in every target, and the only thing that changes is the backing vault or the value CI injects. Each target deploys with its own service principal, so dev can't read prod even by accident.

One more thing worth keeping an eye on: UC external secrets, where a schema is backed directly by AWS Secrets Manager or Azure Key Vault, is in Beta right now. We haven't moved production to it yet but that is where I expect this to land.

Hope it helps, happy to share more detail on the CI side if useful.

Re: Managing "Secret" Injection in DABs across Dev/Stage/Prod

Islam_hoti — Mon, 28 Sep 2026 09:36:50 GMT

Hi,

The basic rule: the bundle manages secret scopes and references to secrets, never the secret values.

1. Keeping secrets out of databricks.yml: Only reference scope and key names. Jobs resolve the secret at runtime:

In code: dbutils.secrets.get(scope, key)
In cluster Spark config or environment variables: {{secrets/<scope>/<key>}}

Make the scope name a bundle variable so each target uses its own scope:

yaml
variables:
secret_scope:
default: ingest-dev

targets:
prod:
variables:
secret_scope: ingest-prod

resources:
jobs:
ingest:
tasks:
- task_key: load
new_cluster:
spark_env_vars:
API_KEY: "{{secrets/${var.secret_scope}/api-key}}"

The value is resolved on the cluster and redacted in logs. Nothing sensitive goes into Git or the deployment state.

You can also inject values through BUNDLE_VAR_* environment variables from your CI secrets. That works, but the resolved value ends up in the bundle's deployment state, so keep it for non-sensitive config.

2. Secret scopes vs. an external vault:

Azure: use Key Vault-backed secret scopes. Secrets live in Key Vault and are read live, so rotating a secret needs no redeploy. These scopes are read-only from Databricks, so manage the values in Key Vault.
AWS/GCP: use Databricks-backed scopes, filled by your CI pipeline from Secrets Manager or Secret Manager (for example databricks secrets put-secret) as a separate step from bundle deploy.

3. Environment parity: Since CLI v0.252, bundles support a secret_scopes resource, so the scope is defined once and deployed to every target:

yaml
resources:
secret_scopes:
ingest:
name: ingest-${bundle.target}
backend_type: AZURE_KEYVAULT
keyvault_metadata:
resource_id: ${var.kv_resource_id}
dns_name: ${var.kv_dns_name}
permissions:
- group_name: data-engineers
level: READ

Give each environment its own Key Vault with the same secret names. Only kv_resource_id and kv_dns_name change per target, so the code is identical in dev, qa and prod.

Two notes:

Creating a Key Vault-backed scope requires the deploying identity to authenticate with Microsoft Entra ID. A PAT fails with a userAADToken error.
Scopes that already exist aren't managed by the bundle automatically. Import them or recreate them through the bundle.

Docs: Bundle resources · Secret Scope API · Secret management (Azure) · KB: userAADToken error