10-17-2022 06:39 AM
I have a few catalogs in databricks (using Unity Catalog). Each catalog (DEV, TEST, PROD) having its own external location (ADLSv2) and being tied to its own dbx workspace. Seeing how unity unifies the data, access policies and the metastore I would assume I could query data in PROD from DEV and vice versa.
However, I am getting this error when trying to do so:
-- in dev environment
USE CATALOG prod;
SELECT * FROM bronze.demo;
>> Error in SQL statement: ExecutionException: Status code: -1 error code: null error message: Cannot resolve hostname: xxxxPRODxxx.dfs.core.windows.net
java.net.UnknownHostException: xxxxPRODxxx.dfs.core.windows.net
10-17-2022 07:20 AM
@Oscar Dyremyhr It looks issue is not specific to unity catalog, but DNS resolution, can you try below steps from your unity catalog configured cluster by following below steps in article
UnknownHostException on cluster launch - Azure Databricks | Microsoft Learn
10-17-2022 07:40 AM
@karthik p The issue is not upon cluster launch, although it might be related. I am still able to access data from DEV. I will take a look.
10-17-2022 11:33 PM
Hi @Oscar Dyremyhr , looking at the error it looks like it is not able to resolve the hostname, could you please try to nslookup the hostname and see if it is getting resolved? Also, what is used in your connection string? ("account" can be used)
10-24-2022 02:43 AM
@Debayan Mukherjee Thanks for the feedback. This solved my issue. Had to add some private endpoints to the underlying storage accounts in where the catalog was referencing data from.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.