4 weeks ago
We have Azure Databricks with standard private link (back-end and front-end private link).
We are able to successfully attach a Databricks workspace to the Databricks metastore (ADLS Gen2 storage).
However, when trying to create tables in a catalog in the Databricks metastore, running from a cluster on the Databricks workspace, I run into the following scenario:
It seems like we are close to getting this to work. Do we need to allow traffic to that external IP, even with standard private link? Any ideas on what might be going on?
Thanks!
4 weeks ago
@m997al
You still need to whitelist some of the IPs on your firewall. This can be done through service tags:
https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/udr
4 weeks ago
Thanks @daniel_sahal ! So we are trying to get the full list of what we need to whitelist.
The Microsoft Azure documentation is a little unclear for what we need specifically, have Azure Databricks standard private link and SCC ("No Public IP" for the clusters).
I did find this:
...and those in turn tie to these URLs...
... I see some URLs for "Artifact Blob storage secondary" and "System tables storage" that are not referenced in the first list... do we need those too?
Thanks for your help!
3 weeks ago
@m997al
Yes, they are needed too.
Basically Service Tag is a bundled list of IPs, so if you're using Azure Firewall, you don't need to put each of one separately, you can just use service tag.
If you're using your own Firewall, then you need to whitelist each of IP provided in documentation.
NOTE: If you want to see which IPs Service Tag contains, here is a full list: https://www.microsoft.com/en-us/download/details.aspx?id=56519
3 weeks ago
Great, thank you!
3 weeks ago
can confirm that the approach will solve your error. Ran into a similar issue a while back.
3 weeks ago
Thank you!