cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Using a custom Hostname in Databricks CLI instead of per-workspace URL

NadithK
Contributor
Hi,

At our organization, we have added front end privatelink connection to a Databricks workspace in Azure, and public access to the workspace is disabled. I am able to access the workspace UI with the private IP (in the browser), and able to call the REST APIs. But I am unable to access the workspace using Databricks CLI. Getting below error when I try to connect.

Error: SSLError: HTTPSConnectionPool(host='10.81.x.x', port=443): Max retries exceeded with url: /api/2.0/clusters/list (Caused by SSLError(CertificateError("hostname '10.81.x.x' doesn't match either of '*.azuredatabricks.net', '*.0.azuredatabricks.net', '*.1.azuredatabricks.net', '*.2.azuredatabricks.net', '*.3.azuredatabricks.net', '*.4.azuredatabricks.net', '*.5.azuredatabricks.net', '*.6.azuredatabricks.net', '*.7.azuredatabricks.net', '*.8.azuredatabricks.net', '*.9.azuredatabricks.net', '*.10.azuredatabricks.net', '*.11.azuredatabricks.net', '*.12.azuredatabricks.net', '*.13.azuredatabricks.net', '*.14.azuredatabricks.net', '*.15.azuredatabricks.net', '*.16.azuredatabricks.net', '*.17.azuredatabricks.net', '*.18.azuredatabricks.net', '*.19.azuredatabricks.net', 'azuredatabricks.net', '0.azuredatabricks.net', '1.azuredatabricks.net', '2.azuredatabricks.net', '3.azuredatabricks.net', '4.azuredatabricks.net', '5.azuredatabricks.net', '6.azuredatabricks.net', '7.azuredatabricks.net', '8.azuredatabricks.net', '9.azuredatabricks.net', '10.azuredatabricks.net', '11.azuredatabricks.net', '12.azuredatabricks.net', '13.azuredatabricks.net', '14.azuredatabricks.net', '15.azuredatabricks.net', '16.azuredatabricks.net', '17.azuredatabricks.net', '18.azuredatabricks.net', '19.azuredatabricks.net', '*.pl-auth.azuredatabricks.net', 'pl-auth.azuredatabricks.net'")))

My .databrickscfg file looks like below.
[DEFAULT]
host = https://10.81.x.x/
token = xxxxxxxxxxxxxxxxxxxxxxxx
jobs-api-version = 2.0
Can someone help me how I could resolve this. We are planning to replace these 10.81.x.x private IPs with custom hostnames down the line using our internal DNS.

Thank you.
2 ACCEPTED SOLUTIONS

Accepted Solutions

Kaniz_Fatma
Community Manager
Community Manager

Hi @NadithKThe error you're experiencing is related to SSL certificate validation. When using Databricks CLI, it attempts to validate the SSL certificate of the endpoint it connects to. In your case, it's trying to validate the IP address ’10.81.x.x’ certificate, but it is valid for ’*.azuredatabricks.net’ domains and not for the IP address. As you mentioned, using your internal DNS, you plan to replace these private IPs with custom hostnames. Once you have the custom hostnames, update the .databrickscfg file to use the custom hostname instead of the IP address. The hostname should match the pattern ’*.azuredatabricks.net’ to pass the SSL certificate validation.

View solution in original post

Hi @NadithKNo, using a custom hostname like .adb<my organization>.net would not work. Azure Databricks requires specific DNS configuration and the use of particular hostnames, typically in the format of .azuredatabricks.net.The information provided indicates that UnknownHostException errors, often caused by DNS configuration issues, can occur when launching an Azure Databricks cluster. These errors can be caused by problems such as the primary DNS being down or unresponsive, artefacts not being resolved, or a host record listing the artefact public IP as static when it has changed. To resolve these issues, the solution suggests identifying a working DNS server and updating the DNS entry on the cluster, verifying the reachability of the artefacts blob storage account and the primary DNS server, and updating the nameserver value with a working DNS server. Therefore, using a custom hostname like *.adb<my organization>.net would likely result in DNS configuration issues and the inability to launch and operate an Azure Databricks cluster properly. 

Sources:
1. [UnknownHostException on cluster launch](https://kb.databricks.com/clusters/unknown-host-exception-on-launch)
2. [Configure internal DNS to redirect user requests to the web application (for front-end)](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html)
3. [Azure CLI authentication](https://docs.databricks.com/dev-tools/cli/databricks-cli.html)

View solution in original post

5 REPLIES 5

Kaniz_Fatma
Community Manager
Community Manager

Hi @NadithKThe error you're experiencing is related to SSL certificate validation. When using Databricks CLI, it attempts to validate the SSL certificate of the endpoint it connects to. In your case, it's trying to validate the IP address ’10.81.x.x’ certificate, but it is valid for ’*.azuredatabricks.net’ domains and not for the IP address. As you mentioned, using your internal DNS, you plan to replace these private IPs with custom hostnames. Once you have the custom hostnames, update the .databrickscfg file to use the custom hostname instead of the IP address. The hostname should match the pattern ’*.azuredatabricks.net’ to pass the SSL certificate validation.

Hi @Kaniz_Fatma,
Thank you for the reply.
Would this not work if I use a custom hostname without *.azuredatabricks.net and use something like

*.adb<my organization>.net

Hi @NadithKNo, using a custom hostname like .adb<my organization>.net would not work. Azure Databricks requires specific DNS configuration and the use of particular hostnames, typically in the format of .azuredatabricks.net.The information provided indicates that UnknownHostException errors, often caused by DNS configuration issues, can occur when launching an Azure Databricks cluster. These errors can be caused by problems such as the primary DNS being down or unresponsive, artefacts not being resolved, or a host record listing the artefact public IP as static when it has changed. To resolve these issues, the solution suggests identifying a working DNS server and updating the DNS entry on the cluster, verifying the reachability of the artefacts blob storage account and the primary DNS server, and updating the nameserver value with a working DNS server. Therefore, using a custom hostname like *.adb<my organization>.net would likely result in DNS configuration issues and the inability to launch and operate an Azure Databricks cluster properly. 

Sources:
1. [UnknownHostException on cluster launch](https://kb.databricks.com/clusters/unknown-host-exception-on-launch)
2. [Configure internal DNS to redirect user requests to the web application (for front-end)](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html)
3. [Azure CLI authentication](https://docs.databricks.com/dev-tools/cli/databricks-cli.html)

NadithK
Contributor

Hi @Kaniz_Fatma ,

Thank you for the support.
Really appreciate it.

Thanks

Hi @NadithKYou're very welcome! Your success is our motivation. Keep up the great work!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!