02-13-2024 07:23 AM
I'm trying to create storage credentials for an Azure Databricks Connector at the workspace level with a service principal that has the CREATE_STORAGE_CREDENTIAL but is NOT an account admin. For this test, the SP has the owner role on the connector.
I've tried this via Terraform, Databricks CLI and Databricks CLI API with a 500 response (or an error with no message) in all cases.
Simplified Terraform:
resource "azurerm_databricks_access_connector" "managed" {
name = format("ac-%s-%s", local.oldw_environment, local.business_unit)
resource_group_name = data.azurerm_resource_group.oldw.name
location = data.azurerm_resource_group.oldw.location
identity {
type = "SystemAssigned"
}
}
CLI call:
databricks storage-credentials create --json '{"name": "manual_test", "azure_managed_identity": {"access_connector_id": "/subscriptions/$MY_SUB_ID/resourceGroups/$MY_RG/providers/Microsoft.Databricks/accessConnectors/manual-test"}, "skip_validation": true}' -p $MY_PROFILE --debug
CLI response:
15:11:11 INFO start pid=38926 version=0.211.0 args="databricks, storage-credentials, create, --json, {\"name\": \"manual_test\", \"azure_managed_identity\": {\"access_connector_id\": \"/subscriptions/$MY_SUB_ID/resourceGroups/$MY_RG/providers/Microsoft.Databricks/accessConnectors/manual-test\"}, \"skip_validation\": true}, -p, dev, --debug"
15:11:11 DEBUG Loading dev profile from /Users/sgrosen/.databrickscfg pid=38926 sdk=true
15:11:11 INFO Generating AAD token for Service Principal ($SP_ID) pid=38926 sdk=true
15:11:11 DEBUG POST /$TENANT_ID/oauth2/token
> [non-JSON document of 19 bytes]. <http.RoundTripper>
< HTTP/1.1 200 OK
< {
< "access_token": "**REDACTED**",
< "expires_in": "3599",
< "expires_on": "1707775871",
< "ext_expires_in": "3599",
< "not_before": "1707771971",
< "resource": "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d",
< "token_type": "Bearer"
< } pid=38926 sdk=true
15:11:12 DEBUG POST /$TENANT_ID/oauth2/token
> [non-JSON document of 19 bytes]. <http.RoundTripper>
< HTTP/1.1 200 OK
< {
< "access_token": "**REDACTED**",
< "expires_in": "3599",
< "expires_on": "1707775871",
< "ext_expires_in": "3599",
< "not_before": "1707771971",
< "resource": "https://management.core.windows.net/",
< "token_type": "Bearer"
< } pid=38926 sdk=true
15:11:12 DEBUG non-retriable error: pid=38926 sdk=true
15:11:12 DEBUG POST /api/2.1/unity-catalog/storage-credentials
> {
> "azure_managed_identity": {
> "access_connector_id": "/subscriptions/$MY_SUB_ID/resourceGroups/$MY_RG/pro... (56 more bytes)"
> },
> "name": "manual_test",
> "skip_validation": true
> }
< HTTP/2.0 500 Internal Server Error
< {
< "details": [
< {
< "@type": "type.googleapis.com/google.rpc.RequestInfo",
< "request_id": "3e023b35-1fdf-4004-bd87-b5b3ef54e4df",
< "serving_data": ""
< }
< ],
< "error_code": "INTERNAL_ERROR",
< "message": ""
< } pid=38926 sdk=true
Error:
15:11:12 ERROR failed execution pid=38926 exit_code=1 error=
Any ideas on what is going wrong here? Up until this point, the only storage credentials created have been created manually by account admins. Our group is trying to automate this process but running into this problem.
02-13-2024 09:27 PM
Hi @ledbutter , hope you are doing well today!
I have gone through the details and this issue might be related to https://github.com/databricks/cli/issues/1080
Please refer to this for more details: https://github.com/databricks/cli/issues/1108
Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi
02-13-2024 09:27 PM
Hi @ledbutter , hope you are doing well today!
I have gone through the details and this issue might be related to https://github.com/databricks/cli/issues/1080
Please refer to this for more details: https://github.com/databricks/cli/issues/1108
Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi
02-14-2024 08:20 AM
Thanks for the links! I will watch those issues.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group