cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How exactly to create cluster policy via Databricks CLI ?

guangyi
Contributor II

I tried these ways they are all not working:

 

 

Save the json config into a JSON file locally and run 

databricks cluster-policies create --json cluster-policy.json 

Error message: Error: invalid character 'c' looking for beginning of value



Save the json config into a JSON file locally and run

databricks cluster-policies create --json @"cluster-policy.json"

Error message: Error: 'definition' must be supplied.



Try using `definition` and run:

databricks cluster-policies create --definition {"cluster_type":{"type":"fixed","value":"job"}}

Error message: Error: Definition has invalid JSON



Give a more detailed `definition` and run 

databricks cluster-policies create --definition {"spark_conf.spark.databricks.cluster.profile":{"type":"forbidden","hidden":true},"spark_version":{"type":"unlimited","defaultValue":"auto:latest-lts"},"node_type_id":{"type":"unlimited","defaultValue":"Standard_DS3_v2","isOptional":true},"num_workers":{"type":"unlimited","defaultValue":4,"isOptional":true},"azure_attributes.availability":{"type":"unlimited","defaultValue":"SPOT_WITH_FALLBACK_AZURE"},"azure_attributes.spot_bid_max_price":{"type":"fixed","value":100,"hidden":true},"instance_pool_id":{"type":"forbidden","hidden":true},"driver_instance_pool_id":{"type":"forbidden","hidden":true},"cluster_type":{"type":"fixed","value":"job"}}

Error message: Error: accepts 1 arg(s), received 20

1 ACCEPTED SOLUTION

Accepted Solutions

szymon_dybczak
Contributor III

Hi @guangyi ,

Try to add @ before the name of json file

databricks cluster-policies create --json @policy.json

 

Also make sure that you're escaping quotation marks like they do in below documenation:

Create a new policy | Cluster Policies API | REST API reference | Azure Databricks

 

{
  "name": "Test policy",
  "definition": "{ \"custom_tags.test_tag\": { \"type\": \"fixed\", \"value\": \"test_value\" } }\n"
}

 


In my case after these changes I was able to create policy:

Slash_0-1722323175186.png

 



View solution in original post

1 REPLY 1

szymon_dybczak
Contributor III

Hi @guangyi ,

Try to add @ before the name of json file

databricks cluster-policies create --json @policy.json

 

Also make sure that you're escaping quotation marks like they do in below documenation:

Create a new policy | Cluster Policies API | REST API reference | Azure Databricks

 

{
  "name": "Test policy",
  "definition": "{ \"custom_tags.test_tag\": { \"type\": \"fixed\", \"value\": \"test_value\" } }\n"
}

 


In my case after these changes I was able to create policy:

Slash_0-1722323175186.png

 



Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group