topic IP address fix in Data Engineering

IP address fix

Jon — Thu, 21 Oct 2021 07:14:40 GMT

How can I fix the IP address of my Azure Cluster so that I can whitelist the IP address to run my job daily on my python notebook? Or can I find out the IP address to perform whitelisting? Thanks

Re: IP address fix

Hubert-Dudek — Thu, 21 Oct 2021 08:39:14 GMT

You can use pool for instances and set that there is "min idle" = "max capacity" so it will not auto terminate so ip will remain the same but this solution doesn't guarantee that it will not change (for example during upgrade or something)
Preferred solution is when you do VM injection you can set routing and set that traffic through internet is going through firewall which have constant ip address. There is ready Azure firewall but a bit expensive, you can install own firewall on VM (there are ready images on Marketplace).
Some other solution like connection through VPN or proxy requests are possible in some cases as alternative.

Re: IP address fix

-werners- — Thu, 21 Oct 2021 09:56:57 GMT

I would not go for option 1 indeed as Hubert said.

Azure provided solutions (VNET injection etc) are preferred.

The IP range which is used for the Azure VMs (which Databricks uses) will be quite big so whitelisting that whole range defeats the use of a firewall.

Re: IP address fix

pratima_patil — Fri, 05 Apr 2024 10:49:06 GMT

@-werners- Do you think there is any way we can use one single ip to access our target resource from databricks? I know databricks creates/uses so many ip's which is not feasible solution to whitelist each time. Please let me know if you have any idea on it. Thanks.

Re: IP address fix

-werners- — Fri, 05 Apr 2024 11:08:07 GMT

Depends on the scenario. You could expose a single ip address to the external internet, but databricks itself will always use many addresses.