- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-04-2025 08:54 AM
Any thoughts on this? I want to read the default catalog and default schema at runtime and store them in a python variable. I want these values sourced from pipeline settings. spark.conf.getAll() does not work.
Databricks Assistant suggests the following, but this doesn't work either. The error indicates these configs don't exist:
To read the default catalog and default schema from the Lakeflow Declarative Pipeline settings into Python variables, use the following Spark configuration keys:
- spark.databricks.sql.initial.catalog for the default catalog
- spark.databricks.sql.initial.schema for the default schema
Here is how you can assign them to Python variables:
default_catalog = spark.conf.get("spark.databricks.sql.initial.catalog")
default_schema = spark.conf.get("spark.databricks.sql.initial.schema")
These variables will reflect the catalog and schema set in your pipeline configuration. If you want to provide fallback values, you can use the os.getenv approach, but the Spark config is the authoritative source for pipeline settings.
