Using Databricks Connect with PyCharm behind Proxy

ADuma — Wed, 21 Aug 2024 14:11:30 GMT

I am using Databricks connect with PyCharm in order to locally develop my pipelines before deploying to Databricks. I set up authentication for databricks-connect using the following guide https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/pycharm

This is working well when I'm working with a normal internet connection. But when I'm working from my office I enter through a company network which prevents access to many services. I can get around this by specifying the HTTPS_PROXY environment variable for many services (git, pip etc.)

But I don't manage to connect to my Databricks cluster through databricks-connect. I just get the following error message:

ValueError: default auth: databricks-cli: cannot get access token: Error: oidc: fetch .well-known: Get "https://my_workspace.azuredatabricks.net/oidc/.well-known/oauth-authorization-server": proxyconnect tcp: tls: first record does not look like a TLS handshake. Config: host=https://my_workspace.azuredatabricks.net, profile=my-profile, auth_type=databricks-cli, cluster_id=my_cluster_id

Does anyone know how to set the proxy config for databricks-connect? Do I need to set some values in .databrickscfg?

Re: Using Databricks Connect with PyCharm behind Proxy

ADuma — Thu, 29 Aug 2024 09:35:10 GMT

I managed to solve the problem by including my login and password in the HTTPS_PROXY environment variable.

I set it to something like this HTTPS_PROXY = "http://username:password@proxy.company.com:port" and databricks-connect can create spark sessions in my pycharm now.

topic Re: Using Databricks Connect with PyCharm behind Proxy in Administration & Architecture

Using Databricks Connect with PyCharm behind Proxy

Re: Using Databricks Connect with PyCharm behind Proxy