Azure Databricks with VNET injection and SCC

Administration & Architecture

Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.

Hi,

Azure databricks with VNET injection and SCC need to communicate with Azure endpoints for following,

Metastore, artifact Blob storage, system tables storage, log Blob storage, and Event Hubs endpoint IP addresses.

https://learn.microsoft.com/en-us/azure/databricks/resources/ip-domain-region

Couple of questions on above,

1) Data plane clusters need access to Metastore endpoint on port 3306 only if Databricks managed hive metastore is used. If external hive metastore on cloud is used then compute needs connectivity to that metastore? And how about if Unity catalog is used then does compute need to connect on mysql endpoint on port 3306 as defined in above link?

e.g. West Europe

consolidated-westeurope-prod-metastore.mysql.database.azure.com
consolidated-westeurope-prod-metastore-addl-1.mysql.database.azure.com

2) About Event hub endpoints on port 9093, is it must to have this connection open and what sort of use case id behind it?

e.g. prod-westeurope-observabilityeventhubs.servicebus.windows.net

Thanks