Using a custom Hostname in Databricks CLI instead of per-workspace URL

[NadithK]

Hi,

At our organization, we have added front end privatelink connection to a Databricks workspace in Azure, and public access to the workspace is disabled. I am able to access the workspace UI with the private IP (in the browser), and able to call the REST APIs. But I am unable to access the workspace using Databricks CLI. Getting below error when I try to connect.

Error: SSLError: HTTPSConnectionPool(host='10.81.x.x', port=443): Max retries exceeded with url: /api/2.0/clusters/list (Caused by SSLError(CertificateError("hostname '10.81.x.x' doesn't match either of '*.azuredatabricks.net', '*.0.azuredatabricks.net', '*.1.azuredatabricks.net', '*.2.azuredatabricks.net', '*.3.azuredatabricks.net', '*.4.azuredatabricks.net', '*.5.azuredatabricks.net', '*.6.azuredatabricks.net', '*.7.azuredatabricks.net', '*.8.azuredatabricks.net', '*.9.azuredatabricks.net', '*.10.azuredatabricks.net', '*.11.azuredatabricks.net', '*.12.azuredatabricks.net', '*.13.azuredatabricks.net', '*.14.azuredatabricks.net', '*.15.azuredatabricks.net', '*.16.azuredatabricks.net', '*.17.azuredatabricks.net', '*.18.azuredatabricks.net', '*.19.azuredatabricks.net', 'azuredatabricks.net', '0.azuredatabricks.net', '1.azuredatabricks.net', '2.azuredatabricks.net', '3.azuredatabricks.net', '4.azuredatabricks.net', '5.azuredatabricks.net', '6.azuredatabricks.net', '7.azuredatabricks.net', '8.azuredatabricks.net', '9.azuredatabricks.net', '10.azuredatabricks.net', '11.azuredatabricks.net', '12.azuredatabricks.net', '13.azuredatabricks.net', '14.azuredatabricks.net', '15.azuredatabricks.net', '16.azuredatabricks.net', '17.azuredatabricks.net', '18.azuredatabricks.net', '19.azuredatabricks.net', '*.pl-auth.azuredatabricks.net', 'pl-auth.azuredatabricks.net'")))

My .databrickscfg file looks like below.

[DEFAULT]

host = https://10.81.x.x/

token = xxxxxxxxxxxxxxxxxxxxxxxx

jobs-api-version = 2.0

Can someone help me how I could resolve this. We are planning to replace these 10.81.x.x private IPs with custom hostnames down the line using our internal DNS.

Thank you.

[Kaniz · Accepted solution]

Hi @NadithK, The error you're experiencing is related to SSL certificate validation. When using Databricks CLI, it attempts to validate the SSL certificate of the endpoint it connects to. In your case, it's trying to validate the IP address ’10.81.x.x’ certificate, but it is valid for ’*.azuredatabricks.net’ domains and not for the IP address. As you mentioned, using your internal DNS, you plan to replace these private IPs with custom hostnames. Once you have the custom hostnames, update the .databrickscfg file to use the custom hostname instead of the IP address. The hostname should match the pattern ’*.azuredatabricks.net’ to pass the SSL certificate validation.

[NadithK]

Hi @Kaniz,
Thank you for the reply.
Would this not work if I use a custom hostname without *.azuredatabricks.net and use something like

*.adb<my organization>.net

[Kaniz · Accepted solution]

Hi @NadithK, No, using a custom hostname like .adb<my organization>.net would not work. Azure Databricks requires specific DNS configuration and the use of particular hostnames, typically in the format of .azuredatabricks.net.The information provided indicates that UnknownHostException errors, often caused by DNS configuration issues, can occur when launching an Azure Databricks cluster. These errors can be caused by problems such as the primary DNS being down or unresponsive, artefacts not being resolved, or a host record listing the artefact public IP as static when it has changed. To resolve these issues, the solution suggests identifying a working DNS server and updating the DNS entry on the cluster, verifying the reachability of the artefacts blob storage account and the primary DNS server, and updating the nameserver value with a working DNS server. Therefore, using a custom hostname like *.adb<my organization>.net would likely result in DNS configuration issues and the inability to launch and operate an Azure Databricks cluster properly. 

Sources:
1. [UnknownHostException on cluster launch](https://kb.databricks.com/clusters/unknown-host-exception-on-launch)
2. [Configure internal DNS to redirect user requests to the web application (for front-end)](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html)
3. [Azure CLI authentication](https://docs.databricks.com/dev-tools/cli/databricks-cli.html)

[NadithK]

Hi @Kaniz ,

Thank you for the support.
Really appreciate it.

Thanks

[Kaniz]

Hi @NadithK, You're very welcome! Your success is our motivation. Keep up the great work!