2 weeks ago
We have Azure Databricks with standard private link (back-end and front-end private link).
We are able to successfully attach a Databricks workspace to the Databricks metastore (ADLS Gen2 storage).
However, when trying to create tables in a catalog in the Databricks metastore, running from a cluster on the Databricks workspace, I run into the following scenario:
It seems like we are close to getting this to work. Do we need to allow traffic to that external IP, even with standard private link? Any ideas on what might be going on?
Thanks!
a week ago
@m997al
You still need to whitelist some of the IPs on your firewall. This can be done through service tags:
https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/udr
a week ago
Thanks @daniel_sahal ! So we are trying to get the full list of what we need to whitelist.
The Microsoft Azure documentation is a little unclear for what we need specifically, have Azure Databricks standard private link and SCC ("No Public IP" for the clusters).
I did find this:
...and those in turn tie to these URLs...
... I see some URLs for "Artifact Blob storage secondary" and "System tables storage" that are not referenced in the first list... do we need those too?
Thanks for your help!
a week ago
@m997al
Yes, they are needed too.
Basically Service Tag is a bundled list of IPs, so if you're using Azure Firewall, you don't need to put each of one separately, you can just use service tag.
If you're using your own Firewall, then you need to whitelist each of IP provided in documentation.
NOTE: If you want to see which IPs Service Tag contains, here is a full list: https://www.microsoft.com/en-us/download/details.aspx?id=56519
a week ago
Great, thank you!
a week ago
can confirm that the approach will solve your error. Ran into a similar issue a while back.
a week ago
Thank you!
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.