VS Code - ipynb vs py execution - spark issue

[Sisi]

Databricks Connect works inside VS Code notebook but the same code fails in a standalone script with
ValueError: default auth: cannot configure default credentials

I’m developing locally with **Databricks Connect 16.1.6** and VS Code.

Inside a Jupyter notebook(.venv) running one cell works fine:

import sys
sys.path.append("../src")
from test_dab import main
main.get_taxis(spark).show(10)

However, when I am trying to debug the main file (Python Debugger: Debug Python file) I am getting the error:

from pyspark.sql import SparkSession, DataFrame

def get_taxis(spark: SparkSession) -> DataFrame:
    return spark.read.table("samples.nyctaxi.trips")

def get_spark() -> SparkSession:
    try:
        from databricks.connect import DatabricksSession

        return DatabricksSession.builder.getOrCreate()
    except ImportError:
        return SparkSession.builder.getOrCreate()


def main():
    get_taxis(get_spark()).show(5)


if __name__ == "__main__":
    main()​

File "c:\[...]\main.py", line 15, in get_spark return DatabricksSession.builder.getOrCreate()

...

ValueError: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method.

When I run main.py by "Debug current file with Databricks Connect" then it works. 
Why is that?

[szymon_dybczak]

Hi @Sisi ,

I think what's happening here is when you debug with option "Debug current file with Databricks Connect" then VS Code is using Databricks extension, which automatically handles authentication and sets up proper configuration.

The regular Python debugger doesn't have access to the Databricks-specific authentication context that VS Code's extension provides.