Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-19-2025 07:29 AM
Hi @DatabricksEngi1 - Please ensure you have a Python Venv set up for each Python version that you use with Databricks Connect. Also, I have given step-by-step ways to debug the issue, clear the cache, etc [Read the files and instructions carefully before running them]. Here is the code that I tested, switching between serverless and classic compute. And also my profile entry for your reference. Since cluster_id is already there in the profile I do not need to mention that in the code -
from databricks.connect import DatabricksSession
# Option 1: Use cluster_id from .databrickscfg automatically
# Since your [fielddemo] profile has cluster_id configured, just use the profile
spark = DatabricksSession.builder.profile("fielddemo").getOrCreate()
# Option 2: Use serverless compute
# spark = DatabricksSession.builder.profile("fielddemo").serverless().getOrCreate()
df = spark.read.table("dkushari_uc.dais2025.customers_iceberg")
df.show(5)
[fielddemo]
host = https://workspace.cloud.databricks.com/
token = dapiXXXXX
jobs-api-version = 2.0
cluster_id = XXXX-12345-xxxxxx