I'm trying to create storage credentials for an Azure Databricks Connector at the workspace level with a service principal that has the CREATE_STORAGE_CREDENTIAL but is NOT an account admin. For this test, the SP has the owner role on the connector.
I've tried this via Terraform, Databricks CLI and Databricks CLI API with a 500 response (or an error with no message) in all cases.
Simplified Terraform:
resource "azurerm_databricks_access_connector" "managed" {
name = format("ac-%s-%s", local.oldw_environment, local.business_unit)
resource_group_name = data.azurerm_resource_group.oldw.name
location = data.azurerm_resource_group.oldw.location
identity {
type = "SystemAssigned"
}
}
CLI call:
databricks storage-credentials create --json '{"name": "manual_test", "azure_managed_identity": {"access_connector_id": "/subscriptions/$MY_SUB_ID/resourceGroups/$MY_RG/providers/Microsoft.Databricks/accessConnectors/manual-test"}, "skip_validation": true}' -p $MY_PROFILE --debug
CLI response:
15:11:11 INFO start pid=38926 version=0.211.0 args="databricks, storage-credentials, create, --json, {\"name\": \"manual_test\", \"azure_managed_identity\": {\"access_connector_id\": \"/subscriptions/$MY_SUB_ID/resourceGroups/$MY_RG/providers/Microsoft.Databricks/accessConnectors/manual-test\"}, \"skip_validation\": true}, -p, dev, --debug"
15:11:11 DEBUG Loading dev profile from /Users/sgrosen/.databrickscfg pid=38926 sdk=true
15:11:11 INFO Generating AAD token for Service Principal ($SP_ID) pid=38926 sdk=true
15:11:11 DEBUG POST /$TENANT_ID/oauth2/token
> [non-JSON document of 19 bytes]. <http.RoundTripper>
< HTTP/1.1 200 OK
< {
< "access_token": "**REDACTED**",
< "expires_in": "3599",
< "expires_on": "1707775871",
< "ext_expires_in": "3599",
< "not_before": "1707771971",
< "resource": "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d",
< "token_type": "Bearer"
< } pid=38926 sdk=true
15:11:12 DEBUG POST /$TENANT_ID/oauth2/token
> [non-JSON document of 19 bytes]. <http.RoundTripper>
< HTTP/1.1 200 OK
< {
< "access_token": "**REDACTED**",
< "expires_in": "3599",
< "expires_on": "1707775871",
< "ext_expires_in": "3599",
< "not_before": "1707771971",
< "resource": "https://management.core.windows.net/",
< "token_type": "Bearer"
< } pid=38926 sdk=true
15:11:12 DEBUG non-retriable error: pid=38926 sdk=true
15:11:12 DEBUG POST /api/2.1/unity-catalog/storage-credentials
> {
> "azure_managed_identity": {
> "access_connector_id": "/subscriptions/$MY_SUB_ID/resourceGroups/$MY_RG/pro... (56 more bytes)"
> },
> "name": "manual_test",
> "skip_validation": true
> }
< HTTP/2.0 500 Internal Server Error
< {
< "details": [
< {
< "@type": "type.googleapis.com/google.rpc.RequestInfo",
< "request_id": "3e023b35-1fdf-4004-bd87-b5b3ef54e4df",
< "serving_data": ""
< }
< ],
< "error_code": "INTERNAL_ERROR",
< "message": ""
< } pid=38926 sdk=true
Error:
15:11:12 ERROR failed execution pid=38926 exit_code=1 error=
Any ideas on what is going wrong here? Up until this point, the only storage credentials created have been created manually by account admins. Our group is trying to automate this process but running into this problem.
