[ Databricks - Delta sharing ] Issue with Delta Sharing in Databricks: Unable to Query Shared Views

[MAR1]

Hi guys, I've encountered an issue while attempting to query shared views via Delta Sharing in Databricks. We are using Delta Sharing Databricks-to-Databricks protocol to share data from a databricks environment deployed on azure to another databricks environment deployed on aws. We could see the schema and views shared with us but when we tried to query the data we got the following errors:                                                                                          

1. By using a Serverless sql warehouse compute instance we got this error:

• This Azure storage request is not authorized. The storage account's 'Firewalls and virtual networks' settings may be blocking access to storage services. Please verify your Azure storage credentials or firewall exception settings.According to the Databricks documentation, if our Databricks account is different from the provider's, we must use a Serverless SQL warehouse to query shared views. However, serverless SQL warehouses do not have public IP addresses ( Serverless SQL warehouses do not have public IP addresses. (https://docs.databricks.com/en/admin/sql/serverless.html )) , which poses a challenge for IP whitelisting on the Azure storage side. Anyone has an idea on how to fix this ? Thanks !  

I also tried to use this compute instances :

  Using a shared cluster we got this error:

• UnityCatalogServiceException: [RequestId=69be237f-xxxx-4835-xxxx-285afe90a20f ErrorClass=CUSTOMER_UNAUTHORIZED] The current compute is not supported to query a shared view via Delta Sharing. If the view is shared under the same account, you must use a shared cluster or SQL warehouse. Sharing outside of the same account requires a Serverless SQL warehouse.

 Using a sql warehouse pro instance we got this error:

• SparkConnectGrpcException: (shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException) Failed to acquire a SAS token for list on /managed/__unitystorage/catalogs/xxxxx-5819-xxxxx-b7df-e8a8cd5f4308/tables/xxxx-0ffb-48ee-af3f-xxxxx/_delta_log due to java.util.concurrent.ExecutionException: com.databricks.sql.managedcatalog.UnityCatalogServiceException: [RequestId=xxxx-4064-47b2-98d1-af809b80a001 ErrorClass=CUSTOMER_UNAUTHORIZED] The current compute is not supported to query a shared view via Delta Sharing. If the view is shared under the same account, you must use a shared cluster or SQL warehouse. Sharing outside of the same account requires a Serverless SQL warehouse. 

[Kaniz_Fatma]

Hi @MAR1,  It seems you’ve encountered some challenges while working with Delta Sharing in Databricks. Let’s address each of the issues you’ve encountered:

1. SQL Serverless Warehouse and IP Whitelisting:

   • When using a SQL Serverless warehouse to query shared views, you’re right that it doesn’t have a public IP address. This can indeed pose a challenge for IP whitelisting on the Azure storage side.
   • To overcome this, consider the following approaches:
     • Azure Private Link: Set up an Azure Private Link for your storage account. This allows you to access the storage account over a private network without exposing it to the public internet. You can then whitelist the private IP addresses of your Serverless SQL warehouse.
     • Service Endpoints: Configure service endpoints for your storage account. Service endpoints allow traffic from specific subnets (such as your Databricks workspace) to reach the storage account without going through the public internet. Again, you can whitelist these subnets.
     • Managed Private Endpoints: If you’re using Azure Data Lake Storage Gen2, consider creating a managed private endpoint. This provides a private IP address within your VNet for accessing the storage account.
   • Ensure that your Azure storage credentials are correctly configured and that the necessary permissions are granted.

2. Shared Cluster and SQL Warehouse Pro:

   • The error message indicates that the current compute is not supported to query a shared view via Delta Sharing.
   • If the view is shared under the same account, you must use a shared cluster or a SQL warehouse.
   • For sharing outside of the same account, a Serverless SQL warehouse is required.
   • Make sure you’re using the appropriate compute type based on your sharing scenario.

3. Failed SAS Token Acquisition:

   • When using a sql warehouse pro instance, you encountered an issue acquiring an SAS token for listing data in the storage account.
   • Verify that the credentials and permissions are correctly set up for the storage account.
   • Double-check the configuration related to SAS tokens and ensure that the necessary access rights are granted.

Remember to review the documentation and verify your configurations. If you continue to face issues, consider reaching out for further assistance. Good luck! 🚀

For more details, you can refer to the Databricks documentation on Delta Sharing and the Azure Databricks troubleshooting guide.