Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 07:48 AM
I use it and it works (kinda).
What you have to do is define a variable for the compute policy in the variables section (databricks.yml).
In the target section (also databricks.yml) you set the policy ID per environment (dev, prod,...).
Then in your job yaml, you need to call the variable in the cluster definition in the new_cluster section using policy_id:
policy_id: ${var.policy}
Like that the policy will be used for cluster creation.
Mind that you still have to pass certain values even though the policy already contains them (spark_version f.e.)