VS Code - ipynb vs py execution - spark issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā07-09-2025 06:33 AM
Databricks Connect works inside VS Code notebook but the same code fails in a standalone script with
ValueError: default auth: cannot configure default credentials
Iām developing locally with **Databricks Connect 16.1.6** and VS Code.
Inside a Jupyter notebook(.venv) running one cell works fine:
import sys
sys.path.append("../src")
from test_dab import main
main.get_taxis(spark).show(10)from pyspark.sql import SparkSession, DataFrame
def get_taxis(spark: SparkSession) -> DataFrame:
return spark.read.table("samples.nyctaxi.trips")
def get_spark() -> SparkSession:
try:
from databricks.connect import DatabricksSession
return DatabricksSession.builder.getOrCreate()
except ImportError:
return SparkSession.builder.getOrCreate()
def main():
get_taxis(get_spark()).show(5)
if __name__ == "__main__":
main()āFile "c:\[...]\main.py", line 15, in get_spark return DatabricksSession.builder.getOrCreate()
...
ValueError: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method.
When I run main.py by "Debug current file with Databricks Connect" then it works.
Why is that?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā07-09-2025 07:11 AM
Hi @Sisi ,
I think what's happening here is when you debug with option "Debug current file with Databricks Connect" then VS Code is using Databricks extension, which automatically handles authentication and sets up proper configuration.
The regular Python debugger doesn't have access to the Databricks-specific authentication context that VS Code's extension provides.