08-24-2023 12:57 AM
08-24-2023 03:11 AM
Hi @NadithK, The error you're experiencing is related to SSL certificate validation. When using Databricks CLI, it attempts to validate the SSL certificate of the endpoint it connects to. In your case, it's trying to validate the IP address ’10.81.x.x’ certificate, but it is valid for ’*.azuredatabricks.net’ domains and not for the IP address. As you mentioned, using your internal DNS, you plan to replace these private IPs with custom hostnames. Once you have the custom hostnames, update the .databrickscfg file to use the custom hostname instead of the IP address. The hostname should match the pattern ’*.azuredatabricks.net’ to pass the SSL certificate validation.
08-25-2023 02:04 AM
Hi @NadithK, No, using a custom hostname like .adb<my organization>.net would not work. Azure Databricks requires specific DNS configuration and the use of particular hostnames, typically in the format of .azuredatabricks.net.The information provided indicates that UnknownHostException errors, often caused by DNS configuration issues, can occur when launching an Azure Databricks cluster. These errors can be caused by problems such as the primary DNS being down or unresponsive, artefacts not being resolved, or a host record listing the artefact public IP as static when it has changed. To resolve these issues, the solution suggests identifying a working DNS server and updating the DNS entry on the cluster, verifying the reachability of the artefacts blob storage account and the primary DNS server, and updating the nameserver value with a working DNS server. Therefore, using a custom hostname like *.adb<my organization>.net would likely result in DNS configuration issues and the inability to launch and operate an Azure Databricks cluster properly.
Sources:
1. [UnknownHostException on cluster launch](https://kb.databricks.com/clusters/unknown-host-exception-on-launch)
2. [Configure internal DNS to redirect user requests to the web application (for front-end)](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html)
3. [Azure CLI authentication](https://docs.databricks.com/dev-tools/cli/databricks-cli.html)
08-24-2023 03:11 AM
Hi @NadithK, The error you're experiencing is related to SSL certificate validation. When using Databricks CLI, it attempts to validate the SSL certificate of the endpoint it connects to. In your case, it's trying to validate the IP address ’10.81.x.x’ certificate, but it is valid for ’*.azuredatabricks.net’ domains and not for the IP address. As you mentioned, using your internal DNS, you plan to replace these private IPs with custom hostnames. Once you have the custom hostnames, update the .databrickscfg file to use the custom hostname instead of the IP address. The hostname should match the pattern ’*.azuredatabricks.net’ to pass the SSL certificate validation.
08-25-2023 12:52 AM
Hi @Kaniz,
Thank you for the reply.
Would this not work if I use a custom hostname without *.azuredatabricks.net and use something like
*.adb<my organization>.net
08-25-2023 02:04 AM
Hi @NadithK, No, using a custom hostname like .adb<my organization>.net would not work. Azure Databricks requires specific DNS configuration and the use of particular hostnames, typically in the format of .azuredatabricks.net.The information provided indicates that UnknownHostException errors, often caused by DNS configuration issues, can occur when launching an Azure Databricks cluster. These errors can be caused by problems such as the primary DNS being down or unresponsive, artefacts not being resolved, or a host record listing the artefact public IP as static when it has changed. To resolve these issues, the solution suggests identifying a working DNS server and updating the DNS entry on the cluster, verifying the reachability of the artefacts blob storage account and the primary DNS server, and updating the nameserver value with a working DNS server. Therefore, using a custom hostname like *.adb<my organization>.net would likely result in DNS configuration issues and the inability to launch and operate an Azure Databricks cluster properly.
Sources:
1. [UnknownHostException on cluster launch](https://kb.databricks.com/clusters/unknown-host-exception-on-launch)
2. [Configure internal DNS to redirect user requests to the web application (for front-end)](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html)
3. [Azure CLI authentication](https://docs.databricks.com/dev-tools/cli/databricks-cli.html)
08-26-2023 07:00 PM
Hi @Kaniz ,
Thank you for the support.
Really appreciate it.
Thanks
08-27-2023 10:01 PM
Hi @NadithK, You're very welcome! Your success is our motivation. Keep up the great work!