Thanks for your time @Debayan Mukherjee​  and @Kaniz Fatma​ . We have figured out the issue along with the infra team that we had to do a public ip whitelisting of the databricks clusters in azure.

I have checked the ip adress from the Spark cluster UI - Master but that was private one.

So I used the below code to found my public ip.

from requests import get

ip = get('https://api.ipify.org').text

print('My public IP address is:', ip)

The confusion happend here was because the error from databricks was so generic. May be databricks should add some extra exception during when an index-url is not reachable to databricks.  I would say to improve the exception.

the next problem is that the public ips set for these clusters are dynamic, so the ips get changed automatically. so we are finding a way to make it static and whitelist the ips, thanks