topic Re: policy_id in databricks asset bundle workflow in Administration & Architecture

policy_id in databricks asset bundle workflow

Aria — Fri, 02 Aug 2024 02:11:28 GMT

We are using databricks asset bundle for code deployment and biggest issue I am facing is that policy_id is different in each environment.I tried with environment variable sin azure devops and also with declaring the variables in databricks.yaml and then used it in resources folder. However nothing worked till now.

@policy_id @DAB

Re: policy_id in databricks asset bundle workflow

maikl — Thu, 07 Nov 2024 11:59:03 GMT

Hi Aria,

did you solve the issue? or did you use any workaround?

Thank you.

Re: policy_id in databricks asset bundle workflow

-werners- — Thu, 07 Nov 2024 15:48:28 GMT

I use it and it works (kinda).
What you have to do is define a variable for the compute policy in the variables section (databricks.yml).
In the target section (also databricks.yml) you set the policy ID per environment (dev, prod,...).
Then in your job yaml, you need to call the variable in the cluster definition in the new_cluster section using policy_id:
policy_id: ${var.policy}
Like that the policy will be used for cluster creation.
Mind that you still have to pass certain values even though the policy already contains them (spark_version f.e.)

Re: policy_id in databricks asset bundle workflow

maikl — Thu, 07 Nov 2024 18:17:47 GMT

When you define the policy_id in the target block then the variable doesn't make sense. For me works these:

I solved it for now, to override the value for policy_id in the target part:

targets:

dev:

workspace:

host: https://databricks-workspace-a.azuredatabricks.net

root_path: ~/DATABRICKS_BUNDLES

resources:

jobs:

read_data_lake:

name: read_data_lake

job_clusters:

- job_cluster_key: Job_cluster

new_cluster:

policy_id: <POLICY_ID from databricks-workspace-a>

tst:

workspace:

host: https://databricks-workspace-b.azuredatabricks.net

root_path: ~/DATABRICKS_BUNDLES

resources:

jobs:

read_data_lake:

name: read_data_lake

job_clusters:

- job_cluster_key: Job_cluster

new_cluster:

policy_id: <POLICY_ID from databricks-workspace-b>

I think it's useful for few jobs. When you have more then It's useless, because you must define each job in the target block to use your policy_id. If I missed something or am wrong, please let me know 🙂

Re: policy_id in databricks asset bundle workflow

-werners- — Fri, 08 Nov 2024 08:06:08 GMT

Variables are useful but it depends on how you set up the bundles.

I define a policy per target. Since 'policy' is not known to the target, I create a variable and assign it a different value depending on the environment.
This variable is then used in the jobs, which reside in another file.

I keep environment definition and job definition separated so it is easier to promote from dev to prod

Re: policy_id in databricks asset bundle workflow

maikl — Mon, 09 Dec 2024 07:31:38 GMT

Solved by the lookup function 🙂

https://docs.databricks.com/en/dev-tools/bundles/variables.html#retrieve-an-objects-id-value