@Tiwarisk 

Ideally, this should not happen because I also use a similar setup with Private VNET. So whenever the databricks cluster is getting up it gets the IPs within the VNET.  

Please confirm if you have whitelisted the Private VNET IPs instead of the public ones.