VS Code - ipynb vs py execution - spark issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā07-09-2025 06:33 AM
Databricks Connect works inside VS Code notebook but the same code fails in a standalone script with
ValueError: default auth: cannot configure default credentials
Iām developing locally with **Databricks Connect 16.1.6** and VS Code.
Inside a Jupyter notebook(.venv) running one cell works fine:
import sys
sys.path.append("../src")
from test_dab import main
main.get_taxis(spark).show(10)from pyspark.sql import SparkSession, DataFrame
def get_taxis(spark: SparkSession) -> DataFrame:
return spark.read.table("samples.nyctaxi.trips")
def get_spark() -> SparkSession:
try:
from databricks.connect import DatabricksSession
return DatabricksSession.builder.getOrCreate()
except ImportError:
return SparkSession.builder.getOrCreate()
def main():
get_taxis(get_spark()).show(5)
if __name__ == "__main__":
main()āFile "c:\[...]\main.py", line 15, in get_spark return DatabricksSession.builder.getOrCreate()
...
ValueError: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method.
When I run main.py by "Debug current file with Databricks Connect" then it works.
Why is that?