topic Re: Databricks (GCP) Cluster not resolving Hostname into IP address in Administration & Architecture

Databricks (GCP) Cluster not resolving Hostname into IP address

Sadam97 — Thu, 27 Jun 2024 09:56:09 GMT

we have #mongodb hosts that must be resolved to private internal loadbalancer ips ( of another cluster ), and that we are unable to add host aliases in the Databricks GKE cluster in order for the spark to be able to connect to a mongodb and resolve the mongodb hostname into ip address. Databricks workspace is in VPC as mongodb cluster and has subnet access as we are able to telnet the mongodb node ip from databricks noteboo

Getting error, Timed out after 30000 ms while waiting for a server that matches .

Re: Databricks (GCP) Cluster not resolving Hostname into IP address

NandiniN — Tue, 05 Nov 2024 08:32:16 GMT

Hi @Sadam97 ,

As you are able to telnet to the MongoDB node's IP from the Databricks notebook, it means that network connectivity exists between Databricks and the MongoDB cluster. This confirms that the issue is likely related to DNS resolution or how MongoDB hosts are being addressed, rather than network access itself.

From a Databricks notebook or the GKE cluster, run a DNS query (e.g., nslookup or dig) to ensure the MongoDB hostnames resolve to the correct internal IP addresses. If DNS fails, you may need to configure a DNS server in your GKE cluster that can resolve the MongoDB private addresses.

Thanks!

Re: Databricks (GCP) Cluster not resolving Hostname into IP address

NandiniN — Tue, 05 Nov 2024 08:32:29 GMT

However, there is something else that I would like you to quickly try - using a single-user cluster, and a shared access mode cluster. Do you observe the issue only in shared cluster?

Re: Databricks (GCP) Cluster not resolving Hostname into IP address

NandiniN — Wed, 06 Nov 2024 03:52:10 GMT

Also found this - https://community.databricks.com/t5/data-engineering/why-i-m-getting-connection-timeout-when-connecting-to-mongodb/m-p/14868