topic Re: Configure cluster single node with service principal in Azure in Data Engineering

Configure cluster single node with service principal in Azure

wallystart — Thu, 12 Dec 2024 19:22:40 GMT

Hi! We can configure a cluster single node with single user as service principal using this command:

databricks clusters create --json ' { "cluster_name": "my-cluster", "spark_version": "13.3.x-scala2.12", "node_type_id": "Standard_DS3_v2", "driver_node_type_id": "Standard_DS3_v2", "single_user_name": "<CLIENTID>", "data_security_mode": "SINGLE_USER", "runtime_engine": "STANDARD", "num_workers": 0, "spark_conf": { "spark.databricks.cluster.profile":"singleNode", "spark.master":"local[*, 4]" }, "custom_tags": { "ResourceClass":"SingleNode" } } '

But when we want to use this cluster in a notebook we get this error:

Single-user check failed: user '<reserved>@<reserved>.com' attempted to run a command on single-user cluster <CLUSTERID>, but the single user of this cluster is '<CLIENTID>'

We want to get a cluster with a service principal because we know that multiple people may use the cluster with the correct configuration. What we don't know is that it may be missing configuration. In Settings -> Identity and Access -> Service Principal -> Permissions we are managers and users.

If anyone knows we would appreciate it

Re: Configure cluster single node with service principal in Azure

Walter_C — Thu, 12 Dec 2024 19:48:42 GMT

Hello has this user been given with the CAN_BIND permission?

The manager role for SP does not allow users to run notebooks on behalf this clusters as this is part of private preview, to make if work you need to grant can bind permission as follow:

$ vim grant-service-principal.json { "access_control_list": [ { "user_name": "{username}", "permission_level": "CAN_BIND" } ] } $ curl -X PATCH {DATABRICKS_HOST}/api/2.0/permissions/service-principals/{SP_ID} --header "Content-type: application/json" --header "Authorization: Bearer ${DATABRICKS_TOKEN}" --data @Grant-service-principal.json

Re: Configure cluster single node with service principal in Azure

wallystart — Mon, 16 Dec 2024 21:36:24 GMT

Hello Walter_C! Thanks for your answer

We ran this command but got the error "ENDPOINT_NOT_FOUND". Do we need to activate any preview function? Do we have any mistake?

Thanks

Re: Configure cluster single node with service principal in Azure

Walter_C — Mon, 16 Dec 2024 23:17:19 GMT

Can you share an screenshot of the cluster on the UI side please, if you are creating a cluster assigned to a service principal this means that it should have already the Private preview enabled.

If the private preview is not enabled it will throw the error message you got.