Hi,
I have created a new databricks workspace in Azure with backend private link.
Settings are
- Required NSG rules - No Azure Databricks Rule
NSG rules for AAD and azfrontdoor were added as per documentation.
Private endpoint with subresource databricks_ui_api inside private dns zone "privatelink.azuredatabricks.net" was created successfully and I can see DNS record created correctly.
However, when I try to create a compute cluster it fails with below error.
I think it is happening because our DNS zone is created in a separate VNET and Subscription. There is peering between dns zone vnet and workspace vnet but not sure what else needs to be done to make dns resolution work.
Message
Failed to add 2 workers to the compute. Will attempt retry: false. Reason: Control Plane Request Failure
Help
Failed to get instance bootstrap steps from the Databricks Control Plane. Please check that instances have connectivity to the Databricks Control Plane.
Instance bootstrap failed command: GetRunbook
Followed by DNS resolution which does not look right.
