topic Re: Error while establishing JDBC connection to Azure databricks via HTTP proxy in Data Engineering

Error while establishing JDBC connection to Azure databricks via HTTP proxy

jonasmin — Thu, 02 Jun 2022 15:32:31 GMT

I am using the databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure databricks.

The connection needs to be routed through a HTTP proxy. I found parameters that can be configured for using the HTTP proxy:By passing invalid parameters I found that the parameters are parsed by the driver.

Still I see the error:

java.sql.SQLException: [Databricks][DatabricksJDBCDriver](700120) Host ....azuredatabricks.net cannot be resolved through DnsResolver com.databricks.client.jdbc.rpc.InternalDnsResolver. Error Message: No such host is known (....azuredatabricks.net)
       at com.databricks.client.jdbc.rpc.CustomDnsResolverLoader.getResolvedHost(Unknown Source)
       at com.databricks.client.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
       at com.databricks.client.spark.jdbc.DownloadableFetchClientFactory.createClient(Unknown Source)
       at com.databricks.client.hivecommon.core.HiveJDBCCommonConnection.connectToServer(Unknown Source)
       at com.databricks.client.spark.core.SparkJDBCConnection.connectToServer(Unknown Source)
       at com.databricks.client.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
       at com.databricks.client.spark.core.SparkJDBCConnection.establishConnection(Unknown Source)
       at com.databricks.client.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
       at com.databricks.client.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
       at com.databricks.client.jdbc.common.AbstractDriver.connect(Unknown Source)
       at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55)
       at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
       at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:115)
       at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:665)
       at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:544)
       at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753)
       at cloud.celonis.connector.jdbc.services.DatabaseConnectionService.lambda$getConnection$0(DatabaseConnectionService.java:47)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

So, it seems that the proxy is not used after all.

Do you have any suggestions, what to check and how to make the driver use the proxy? Thank you!

Re: Error while establishing JDBC connection to Azure databricks via HTTP proxy

Prabakar — Fri, 03 Jun 2022 09:22:10 GMT

@Jonas Minning the error message says there is a problem with the DNSresolver. Are you using the default DNS or a custom DNS? Suspect the issue is on the DNS resolution. It would be worth checking with your networking team. If it's a custom DNS, you can try changing it to default and check if this works.

Re: Error while establishing JDBC connection to Azure databricks via HTTP proxy

jonasmin — Fri, 03 Jun 2022 09:55:10 GMT

Thanks for your answer @Prabakar Ammeappin ,

the thing is that I need to use the proxy, so I would expect the Driver to connect to the proxy first, e.g. resolving the hostname of the proxy instead of the hostname of the databricks host.

Re: Error while establishing JDBC connection to Azure databricks via HTTP proxy

User16764241763 — Mon, 06 Jun 2022 03:32:05 GMT

We don't think the DNS resolution traffic will go through the Proxy first. Can you try setting below in the host file and see if it connects.

xxxxxxxxxx.azuredatabricks net <IP address>

Ultimately you have to ensure your custom DNS servers are able to resolve the host names without any issues.

Re: Error while establishing JDBC connection to Azure databricks via HTTP proxy

jonasmin — Tue, 07 Jun 2022 07:43:43 GMT

Thanks for the answer @Arvind Ravish,

We added the IP to the local host file.

Then the DNS resolver error disappeared and we got a timeout error instead.

This timeout is probably due to firewall which does not allow direct connection from our client to Azure. So, I think we still are facing the issue that the driver does not seem to use the proxy but tries a direct connection instead.

So, I was wondering which conditions need to be fulfilled for the driver to use the proxy. Do you know something around this?

Re: Error while establishing JDBC connection to Azure databricks via HTTP proxy

Ravikant — Tue, 02 Aug 2022 07:38:22 GMT

Hi @Jonas Minning, were you able to find the solution for this issue? We are also facing similar issue.

Re: Error while establishing JDBC connection to Azure databricks via HTTP proxy

jonasmin — Tue, 02 Aug 2022 08:25:18 GMT

Hi Ravikant,

unfortunately, we have not found another solution than not using a proxy.

Best, Jonas

Re: Error while establishing JDBC connection to Azure databricks via HTTP proxy

MS_Varma — Thu, 04 May 2023 12:45:51 GMT

Hi @Jonas Minning , actually I am also having the same issue and when i looked into the driver related documentation I found that the driver currently only supports SOCKS proxies and I believe this is the reason why we are getting this error. So, I wanted to check if there are any other drivers that support HTTP proxies. If anyone has information on such JDBC driver could you please let me know?

The driver that i am currently using is DatabricksJDBC42-2.6.29

Best, Varma