We have setup a Databricks workspace in our own Azure VNET, including a private endpoint. Connecting to the WS works fine (through the private ip address). However, when creating my first cluster, I run into this problem:
"ADD_NODES_FAILED
...
Failed to get instance bootstrap steps from the Databricks Control Plane. Please check that instances have connectivity to the Databricks Control Plane.
...
and then further on:
Non-authoritative answer: adb-3485221775054017.17.azuredatabricks.net canonical name = adb-3485221775054017.17.privatelink.azuredatabricks.net. adb-3485221775054017.17.privatelink.azuredatabricks.net canonical name = westeurope-c2.azuredatabricks.net. Name: westeurope-c2.azuredatabricks.net Address: 40.74.30.80."
Now, what we haven't yet done is setup an entry in our private DNS zone (we need to setup privatelink.azuredatabricks.net on our site, which will be done tomorrow morning hopefully).
Question: Is it true that the data plane is running IN our VNET and thus, when DNS has been fixed, it will pick up the right ip address and provisioning of the cluster will work?
(just so I understood the concepts properly).
Kind Regards,
Arjen
