VS Code - ipynb vs py execution - spark issue

Sisi — Wed, 09 Jul 2025 13:33:16 GMT

Databricks Connect works inside VS Code notebook but the same code fails in a standalone script with
ValueError: default auth: cannot configure default credentials

I’m developing locally with **Databricks Connect 16.1.6** and VS Code.

Inside a Jupyter notebook(.venv) running one cell works fine:

import sys sys.path.append("../src") from test_dab import main main.get_taxis(spark).show(10)

However, when I am trying to debug the main file (Python Debugger: Debug Python file) I am getting the error:

from pyspark.sql import SparkSession, DataFrame def get_taxis(spark: SparkSession) -> DataFrame: return spark.read.table("samples.nyctaxi.trips") def get_spark() -> SparkSession: try: from databricks.connect import DatabricksSession return DatabricksSession.builder.getOrCreate() except ImportError: return SparkSession.builder.getOrCreate() def main(): get_taxis(get_spark()).show(5) if __name__ == "__main__": main()

File "c:\[...]\main.py", line 15, in get_spark return DatabricksSession.builder.getOrCreate()

...

ValueError: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method.

When I run main.py by "Debug current file with Databricks Connect" then it works.
Why is that?

Re: VS Code - ipynb vs py execution - spark issue

szymon_dybczak — Wed, 09 Jul 2025 14:11:59 GMT

Hi @Sisi ,

I think what's happening here is when you debug with option "Debug current file with Databricks Connect" then VS Code is using Databricks extension, which automatically handles authentication and sets up proper configuration.

The regular Python debugger doesn't have access to the Databricks-specific authentication context that VS Code's extension provides.

topic Re: VS Code - ipynb vs py execution - spark issue in Administration & Architecture

VS Code - ipynb vs py execution - spark issue

Re: VS Code - ipynb vs py execution - spark issue