Hi,

I am using Databricks CLI 0.227.1 for creating a bundle project to deploy job.

As per this , https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/variables I wanted to have variable-overrides.json to have my variables.

I created a json file variable-overrides.json at .databricks/bundle/dev/variable-overrides.json.

I then referred these variables in the job definition like this:

resources:
jobs:
venu_test_poc1_job:
name: venu_test_poc1_job
tasks:
- task_key: ${var.task_key}
job_cluster_key: ${var.job_cluster_key}
spark_python_task:
python_file: ../src/venu_test_poc1/task.py
job_clusters:
- job_cluster_key: ${var.job_cluster_key}
new_cluster:
spark_version: 15.4.x-scala2.12
node_type_id: Standard_D3_v2
autoscale:
min_workers: 1
max_workers: 3

This is my variable-overrides.json:

When I run databricks bundle validate, I get the below error:

Error: reference does not exist: ${var.job_cluster_key}

As per this, I tried using vars.variablename instead of var.variablename and then the validate command went fine.

But the bundle deploy command threw the below error:

Error: Reference to undeclared resource

on bundle.tf.json line 45, in resource.databricks_job.venu_test_poc1_job.task[0]:
45: "job_cluster_key": "${vars.job_cluster_key}",

A managed resource "vars" "job_cluster_key" has not been declared in the root
module.

Error: Reference to undeclared resource

on bundle.tf.json line 49, in resource.databricks_job.venu_test_poc1_job.task[0]:
49: "task_key": "${vars.task_key}"

A managed resource "vars" "task_key" has not been declared in the root
module.

To get things moving, I had put the variable definition and their assignments in the resource bundle config file databricks.yaml itself. But this doesnt look clean as there will be lot of variables that I need to use and it is best to have them in a json separately.

Pls help me understand how to get the values from variable-overrides.json.

Thanks,

Venu

Venu