Hello ,
We have configured our Databricks environment with private endpoint connections injected into our VNET, which includes two subnets (public and private). We have disabled public IPs and are using Network Security Groups (NSGs) on the subnet, as suggested by Microsoft. Additionally, we have a private endpoint for our Azure Data Lake Storage account, where our tables are created, and this storage is located within the same VNET.
We also utilize a private endpoint for authentication in a separate VNET that has been successfully peered with our main VNET. Currently, our developers are running shared or job compute clusters to create tables by transferring data from Storage Account A to Storage Account B.
However, we are seeing many ingress traffic that we believe should not be occurring. Given that both the cluster and the boths storage accounts are in the same VNET, my understanding is that there should be no costs associated with ingress/egress traffic between these resources.
Could you please provide guidance on whether other teams have encountered similar issues? Additionally, any insights into how we might resolve this unexpected ingress traffic would be greatly appreciated.
Thank you for your assistance.
