Issue on Service Credential creation for Azure Databricks access connector

[Saf4Databricks]

Question: Why I'm getting the following error and how can we fix it?

In step 6 of Create service credentials - Azure Databricks | Microsoft Learn when I enter the resource id of my Azure Databricks access connector, I get the following error:

/subscriptions/a76c7e8f-210d-45e5-8f5e-525015b1c881/resourceGroups/rg_SDP/providers/Microsoft.Databricks/accessConnectors/AcceeConnector4Dbricks is not a valid IAM role ARN. Valid ARNs normally look like arn:aws:iam::<account>:role/<iam-role-name>

But the step 6 of the above linked document says the format should be in the following format:

/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>

And, as shown in image below, I'm using the format suggested by step 6:

[szymon_dybczak]

Hi @Saf4Databricks ,

You created Access Connector in Azure, but your Databricks instance is hosted in AWS. So that's why you're getting this error. 

[Saf4Databricks]

Hi @szymon_dybczak, does that mean, Databricks free version cannot be used with Azure, correct? If so, are there any alternatives?

[Coffee77]

Nothing to do with that. You can use Free Edition in Azure for sure. I guess your problem is you should create your workspace in Azure, not in AWS. Here are more details about it: https://learn.microsoft.com/en-us/azure/databricks/getting-started/free-edition

[szymon_dybczak]

Hi @Saf4Databricks ,

Yes, as of now it's a limitation of Free Edition. So if your free edtion is hosted on AWS then you can configure following storage types:

In Premium edition you can configure different storage types.

[Saf4Databricks]

Hi @szymon_dybczak thank you for providing this info - it will help be in broader context. But, when I select Storage Credential (instead of Service Credential) in my Databricks Free Edition | Databricks on AWS, I'm seeing only following two choices (highlighted below):

[szymon_dybczak · Accepted solution]

Hi  ,

My screenshot was referring to External Locations. But it's the same story with a storage credetials as well. So in Premium workspace you have following options:

And FreeEdition is limited to following options. It's a limitation and you can't do much about this 😕

[Saf4Databricks]

Got it. Thank you @szymon_dybczak for sharing your knowledge.

[SteveOstrowski]

Hi @Saf4Databricks,

The error message is the key clue here. When you enter the Azure Access Connector resource ID and get back:

"is not a valid IAM role ARN. Valid ARNs normally look like arn:aws:iam::<account>:role/<iam-role-name>"

This tells you that your Databricks workspace is hosted on AWS, not Azure. The credential creation form is expecting an AWS IAM role ARN because the workspace runs on AWS infrastructure. An Azure Access Connector resource ID (in the format /subscriptions/.../providers/Microsoft.Databricks/accessConnectors/...) is only valid on Azure Databricks workspaces.

As @szymon_dybczak correctly pointed out, the documentation you linked (Create service credentials - Azure Databricks) applies specifically to Azure-hosted Databricks workspaces.

HOW TO DETERMINE YOUR CLOUD PROVIDER

If you are using Databricks Free Edition (community.cloud.databricks.com), that is the AWS-hosted Free Edition. You can confirm this by checking your workspace URL:
- AWS: typically ends in .cloud.databricks.com or .databricks.com with an AWS region
- Azure: typically ends in .azuredatabricks.net

YOUR OPTIONS

1. If you want to use Azure services (like ADLS Gen2 with an Access Connector), you should create your Databricks workspace directly in Azure. Azure offers a Free Edition as well. See the Azure Databricks Free Edition documentation:
https://learn.microsoft.com/en-us/azure/databricks/getting-started/free-edition

2. If you want to stay on the AWS-hosted Free Edition, you would need to use AWS-native credentials. For service credentials on AWS, you create an IAM role and provide its ARN. The documentation for that flow is here:
https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-services/service-credentials

Note that Free Edition does have some limitations on credential types compared to Premium. On AWS Free Edition, storage credentials are limited to AWS-native options (IAM roles).

CREATING SERVICE CREDENTIALS ON AZURE DATABRICKS

For reference, once you have an Azure-hosted workspace, the correct flow for service credentials with an Access Connector is:

1. Create an Azure Databricks Access Connector in the Azure portal
2. Assign the Access Connector's managed identity the appropriate role on your target service
3. In your Azure Databricks workspace, go to Catalog, then External data, then the Credentials tab
4. Click Create credential and select Service Credential
5. Enter the Access Connector resource ID in the format:

/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>

6. If using a user-assigned managed identity, also provide the managed identity resource ID
7. Click Create

Full Azure documentation:
https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-services/service-cred...

I hope this clears up the cloud provider mismatch and helps you move forward.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.