Since we are unable to resolve hostname, it point towards the DNS issue. We can use custom dns using init script and add in the cluster:
%scala
dbutils.fs.put("/databricks/<directory>/dns-masq.sh";,"""
#!/bin/bash
########################################
Configure on-prem dns access.
########################################
sudo apt-get update -y
sudo apt-get install dnsmasq -y --force-yes
## Add dns entries for internal your-company.net name servers
echo server=/databricks.net/10.X.Y.Z | sudo tee --append /etc/dnsmasq.conf
## Find the default DNS settings for the EC2 instance and use them as the default DNS route
ec2_dns=cat /etc/resolv.conf | grep "nameserver"; | cut -d' ' -f 2
echo "Old dns in resolv.conf $ec2_dns"
echo "server=$ec2_dns" | sudo tee --append /etc/dnsmasq.conf
## configure resolv.conf to point to dnsmasq service instead of static resolv.conf file
mv /etc/resolv.conf /etc/resolv.conf.orig
echo nameserver 127.0.0.1 | sudo tee --append /etc/resolv.conf
sudo service dnsmasq restart
""", true)
https://kb.databricks.com/cloud/custom-dns-routing.html
Another way is to use the IP address to connect.
