Injecting Databricks secrets into Databricks Asset Bundles.

[daan_dw]

Hey,

I want to inject Databricks secrets into my Databricks Asset Bundles in order to avoid exposing secrets.
I tried it as shown in the code block below but it gives the error below the code block.
When I hardcode my instance_profile_arn it does work.
How can I inject My Databricks secrets in my Databricks Asset Bundle?
Many thanks!

instance_profile_arn: ${secrets.aws_secrets.cluster_profile_arn}

Error: exit status 1

Error: Reference to undeclared resource

on bundle.tf.json line 42, in resource.databricks_job.running_prd_xml_files_sftp.job_cluster[0].new_cluster.aws_attributes:
42: "instance_profile_arn": "${secrets.aws_secrets.cluster_profile_arn}",

A managed resource "secrets" "aws_secrets" has not been declared in the root
module.

[Vasireddy · Accepted solution]

Hey @daan_dw ,

Possible reason for your problem:

Databricks Asset Bundles use Terraform under the hood, and Terraform cannot resolve Databricks secret references (like ${secrets.aws_secrets.cluster_profile_arn})
at deployment time. Secrets are only accessible at runtime within notebooks and jobs, not during the bundle deployment phase when Terraform is provisioning
resources. This is why you get the "undeclared resource" error - Terraform expects all configuration values to be resolved before creating resources.

Possible Solutions:

Use Bundle Variables with Environment Variables: Define your secret as a variable in databricks.yml and inject it using the BUNDLE_VAR_ prefix during deployment.
Reference it with ${var.variable_name} in your configuration.

CI/CD Platform Secrets (Recommended): Store secrets in your CI/CD platform (GitHub Secrets, Azure DevOps Variables, etc.) and inject them during automated
deployments using environment variables. This keeps secrets secure and outside version control.

Target-Specific Configuration: For non-sensitive values or different environments, define values directly in target sections of your bundle configuration for dev,
staging, and prod environments.

Variable Override Files: Create a local .databricks/bundle/variables.json file (added to .gitignore) for development purposes.

The key is to never reference Databricks secrets directly in bundle configuration and instead use bundle variables that are populated externally at deployment time.