topic Errors using Databricks Extension for VS Code on Windows in Data Engineering

Errors using Databricks Extension for VS Code on Windows

m997al — Tue, 24 Sep 2024 15:23:33 GMT

Hi - I am trying to get my VS Code (running on Windows) to work with the Databricks extension for VS Code. It seems like I can almost get this to work. Here is my setup:

1. Using Databricks Extension v2.4.0
2. Connecting to Databricks cluster with runtime 15.4 LTS ML (Python 3.11)
3. On Windows VS Code, using a virtual environment with Python 3.11.9
4. Using VS Code 1.93.1
5. As part of the Databricks Extension install in VS Code, Databricks Connect 13.3.2 was installed

What works:

1. Connect to cluster is OK
2. Creation of ".bundle/..." workspace folder for temp work is created.
3. Python environment and Databricks Connect seem to have no errors in VS Code
4. Using the "Databricks" icon in VS Code, I am able to "Run current file with Databricks Connect", I seem to be able to run my code! This would be great, except...

What doesn't work:

1. I don't see an easy way to verify that the code actually ran on my Databricks cluster
2. Local environment variables for VS Code are used instead of the environment variables set up for the cluster. This is a big issue. I specifically want to use the environment variables I have set up for the cluster (that use secret scopes)...I want to get away from local environment variables with VS Code
3. Running "databricks-connect test" from the command line in the virtual environment on the Windows machine fails with this error message: \Lib\site-packages\pyspark\sql\session.py", line 488, in getOrCreate raise RuntimeError(RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.
3.1. I am not sure if that even matters, as with the Databricks Extension for VS Code I guess that Databricks Connect is built into the extension, and doesn't need to run from the command line, yet I wouldn't expect this error.
5. Finally, most seriously, when I try to hit "sync" to sync the workspace folder on Databricks ".../.bundle/...." with the local repo folder... the files sync, but I get a "Building..." message that continues forever.
5.1. It looks like the sync completes (by the way, the above is like the 2nd time I ran it, the first time showed all the files synching)...but the "Build..." goes on forever and never stops.

This would be a really great thing for our department if I can get this to work. I want to get rid of anyone using local environment variables, and control everything through Databricks secret scopes referenced through cluster environment variables. It seems like I am close.

Can anyone help? Thanks!

Re: Errors using Databricks Extension for VS Code on Windows

m997al — Mon, 30 Sep 2024 18:14:27 GMT

So I found my problem(s).

I had a local environment variable called "DATABRICKS_HOST" that was set to the wrong URL.
My Databricks runtime version and the databricks-connect version were not the same. When I made them both 15.4.x, everything works as expected.
Using the "Upload and Run File" option of the Databricks extension was exactly what I needed - it uses the environment variables present on the connected Databricks cluster itself (connected to Databricks secret scopes). So that issue was solved as well.

There is still some question around the "sync" that seems to go on forever in VS Code until you cancel the "sync" task in VS Code, but this is not a show-stopper.

Re: Errors using Databricks Extension for VS Code on Windows

Elm8r — Fri, 14 Nov 2025 09:52:02 GMT

Were you able to use your local venv to run the script? I am trying to run a simple script on a cluster using a local env but is not working

Re: Errors using Databricks Extension for VS Code on Windows

Coffee77 — Fri, 14 Nov 2025 10:52:43 GMT

I guess it will be very likely you are already using it but if not, I would suggest to use GIT folders directly in Databricks in conjunction with VS Code as you are doing now. There will be times in which is much faster/straightforward to run notebooks directly in databricks UI that you can synchronize later in VS Code. At least, this is my personal experience, obviously completely subjective 😀