VS Code and Connect - Manual Environment Setup

[spoltier]

The current version of the extension forces usage of certain python packaging / project tools. I used to be able to use the extension with my project which uses pixi, by setting up the dependencies myself in pyproject.toml / pixi.toml.

The current version of the extension ignores that setup completely and starts setting up a project based on uv. This basically makes the connect functionality unusable for my project.

I would like to be able to select the kernel / python environment myself, without the extension making a bunch of changes to my project configuration.

[szymon_dybczak]

Hi @spoltier ,

There's now a neat way to configure this using databricks cli command called: databricks environments setup-local

Check below news:

Automatically create local Python environments compatible with Databricks Runtime : r/databricks

If my answer was helpful, please consider marking it as accepted solution

[spoltier]

Hi @szymon_dybczak ,

This is kind of the feature I'm complaining about. It's nice to have the option, but it's disappointing that the connect extension now refuses to recognize a local env that has sufficient dependencies set up manually (and e.g. doesn't use pip or uv as env managers).

[spoltier]

Reply to follow up - it looks like basic functionality (running cells) works, though the red "Databricks Connect disabled" status indicator is still present.

[mancy34]

I’m running into the same kind of issue with projects that already have their own environment setup. Being able to manually select the Python environment/kernel without the extension forcing uv changes would make Connect much more flexible.

[srini_ve]

@spoltier ,

I’ve come across similar situations where the project already has its own way of managing Python dependencies and environments.

I think a practical approach would be to let the user select the Python interpreter/environment that the project is already using, rather than the extension automatically creating a new uv setup.

For example, if the project is using Pixi, the extension could simply use the Python environment exposed by Pixi for Databricks Connect. The existing pyproject.toml/ pixi.toml could remain untouched.

Having something like “Use existing Python environment” or “Select Python interpreter” in the extension would make Connect much easier to use with projects that already have their dependency management in place.

uv can still be the default option for new projects, but it would be good to have the flexibility to opt out when the project is already configured.

[Islam_hoti]

Databricks Connect is just a Python package, so you can bypass the extension entirely and keep your pixi setup.

Add databricks-connect to your pixi environment, select that interpreter as your kernel in VS Code, and create the session yourself:

from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.profile("my-profile").getOrCreate()

That reads your existing .databrickscfg and connects the same way, without the extension touching pyproject.toml or pixi.toml. You give up the extension features like the cluster picker and running files as jobs, but the Connect part works normally.

For the underlying issue, opening a request on the databricks vscode GitHub repo is probably the right move, since supporting an existing environment instead of provisioning a new one is a reasonable ask.