Establish Cross cloud connectivity between Azure Databricks and AWS s3

[swee]

Hello.

We have the cross cloud configuration set as below:

AWS - VPC, Transit Gateways, AWS Direct Connect

On Premise Data center

Azure - Vnet, Transit Vnet , Express Route.

We are trying to create a Databricks storage credential as below. The AWS IAM objects have been created and validated.

https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials

Error when validating databricks storage credential

Failed - Assume Role
Skipped - Self Assume Role
Skipped - ExternalID Condition

Missing Permissions
Failed to get credentials: the AWS IAM role in the credential is not configured correctly. Please contact your account admin to update the configuration

I believe this could be an error related to the private network setup. Any guidance on what needs to be done to establish the traffic, would be helpful. Thank you.

[Sai_Ponugoti · Accepted solution]

Hello @swee ,
Thank you for your query.

If your storage account is private, you would need to establish a route to that storage account so you can read data.
This is because if your storage is private, your storage account will block access to the public internet.

This GitHub repo contains instructions on how to set up your network configuration based on your requirements.

Once you configure the network configuration I am confident you will be able to read data.

Thank you