Delta Sharing Issue between AWS and Azure

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

We have attempted to setup a delta share between from Azure to AWS.

We can see the delta share table and meta data in AWS, however when we attempt to query the table we hit a problem.

If we use serverless SQL or Notebook and whitelist the IP address from the databricks serverless cluster the query returns expected results.

However, if we attempt to use dedicated IP address from AWS VPC and a non-serverless SQL warehouse and dedicated Compute instances we get the following error:

“HTTP 500 INTERNAL_ERROR

Reason: DS_INTERNAL_ERROR_FROM_DB_DS_SERVER

Endpoint: https://data-sharing.****.internal.azuredatabricks.net:443/api/2.0/delta-sharing/metastores/<metasto...

Trace ID: f92422c0b36284fe3d03aca010de9953”

Any ideas what is stopping the dedicated IP from returning results from the same share?

Thanks in advance for any insights.

Mark