Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-03-2025 08:57 AM
Hi @ashraf1395 - I tried the approach suggested by you and it is not working as expected.
My databricks.yml file has:
bundle:
name: ip_obs_bundle
variables:
v_task_key:
description: The task key.
default: "default_taskkey"
v_job_cluster_key:
description: The cluster key.
default: "default_clusterkey"
v_job_name:
description: The job name.
default: "default_jobname"
include:
- resources/*.yml
My job definition under resources folder:
resources:
jobs:
ip_obs_bundle_job:
name: ${var.v_job_name}
tasks:
- task_key: ${var.v_task_key}
job_cluster_key: ${var.v_job_cluster_key}
spark_python_task:
python_file: ../src/venu_ip_obs_poc1/task.py
job_clusters:
- job_cluster_key: ${var.v_job_cluster_key}
new_cluster:
spark_version: 15.4.x-scala2.12
node_type_id: Standard_D3_v2
autoscale:
min_workers: 1
max_workers: 3
My variables-overrides.json under projectroot/.databricks/bundle/dev looks like:
{
---truncated
"v_job_name": "overrride_jobname",
"v_task_key": "override_taskkey",
"v_job_cluster_key": "override_clusterkey",
--truncated
}
---truncated
"v_job_name": "overrride_jobname",
"v_task_key": "override_taskkey",
"v_job_cluster_key": "override_clusterkey",
--truncated
}
I don't have any environment variable or command line variable that could conflict with variables-overrides.json.
Even after having this, I get to see the jobs and tasks with names that have been taken from default values in the variable definition in databricks.yml, but they don't pick the values from variables-overrides.json.
Can you pls help with this? Not sure what am I missing here.
