VS Code Databricks Connect Cluster Configuration

[Tito]

I am currently setting up the VSCode extension for Databricks Connect, and it’s working fine so far. However, I have a question about cluster configurations. I want to access Unity Catalog from VSCode through the extension, and I’ve noticed that I can only do so when using a cluster with private compute in single-user access mode. When I try to use a cluster with shared compute, it doesn’t seem to work.

Am I missing something, or is this how it’s supposed to function? If so, it would mean that every developer would need to use a single-user cluster, which could lead to many clusters running simultaneously.

[dkushari]

Hi @Tito, you can use the standard cluster (formerly known as the shared cluster) from VSCode using DBConnect. Here is an example,

from databricks.connect import DatabricksSession

# Option 1: Use cluster_id from .databrickscfg automatically
# Since your [fielddemo] profile has cluster_id configured, just use the profile
spark = DatabricksSession.builder.profile("fielddemo").getOrCreate()

# Option 2: Use serverless compute
# spark = DatabricksSession.builder.profile("fielddemo").serverless().getOrCreate()

df = spark.read.table("samples.nyctaxi.trips")
df.show(5)

 Set up your .databrickscfg profile with the cluster_id of the standard cluster. You can obtain the cluster_id from the Cluster Configurations page, as shown below.

[fielddemo]
host             = https://.........cloud.databricks.com/
token            = ..................................
jobs-api-version = 2.0
cluster_id       = ....-.....-cnhxf2p6