- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2025 09:14 AM - edited 05-19-2025 09:15 AM
Hi @sensanjoy
Since ${param} syntax is deprecated in Databricks, here are the best approaches to make minimal changes across your SQL notebooks:
Option 1: Using Python Variables with f-strings (Recommended for minimal changes)
In your param_notebook:
# Set up widgets as before
dbutils.widgets.text("catalog", "catalog_de")
dbutils.widgets.text("schema", "emp")
# Create Python variables that can be used in SQL cells
catalog_name = dbutils.widgets.get("catalog")
schema_name = dbutils.widgets.get("schema")
# Optional: Set spark conf as well for backward compatibility
spark.conf.set("catalog.name", catalog_name)
spark.conf.set("schema.name", schema_name)
In your SQL notebook:
%run ../../config/param_notebook
-- Use Python variables in SQL cells
%sql
SELECT '{catalog_name}' as catalog, '{schema_name}' as schema;
USE {catalog_name};
SELECT * FROM {schema_name}.emp_details;
Option 2: Using spark.sql() with Python f-strings
In your SQL notebook:
%run ../../config/param_notebook
# Execute SQL with f-strings
spark.sql(f"USE {catalog_name}")
spark.sql(f"""
SELECT '{catalog_name}' as catalog, '{schema_name}' as schema
""").display()
spark.sql(f"""
SELECT * FROM {schema_name}.emp_details
""").display()
Option 3: Using Session Variables (Databricks SQL native approach)
In your param_notebook:
dbutils.widgets.text("catalog", "catalog_de")
dbutils.widgets.text("schema", "emp")
# Set session variables
spark.sql(f"SET VAR catalog_name = '{dbutils.widgets.get('catalog')}'")
spark.sql(f"SET VAR schema_name = '{dbutils.widgets.get('schema')}'")
In your SQL notebook:
%run ../../config/param_notebook
-- Use session variables
SELECT ${VAR.catalog_name} as catalog, ${VAR.schema_name} as schema;
USE IDENTIFIER(${VAR.catalog_name});
SELECT * FROM IDENTIFIER(CONCAT(${VAR.schema_name}, '.emp_details'));
Option 4: Hybrid Approach (Best for your current setup)
Keep your param_notebook mostly unchanged:
dbutils.widgets.text("catalog", "catalog_de")
spark.conf.set("catalog.name", dbutils.widgets.get("catalog"))
dbutils.widgets.text("schema", "emp")
spark.conf.set("schema.name", dbutils.widgets.get("schema"))
# Add these lines for the new approach
catalog_name = dbutils.widgets.get("catalog")
schema_name = dbutils.widgets.get("schema")
Update SQL notebooks minimally:
%run ../../config/param_notebook
-- Replace ${catalog.name} with {catalog_name}
SELECT '{catalog_name}' as catalog, '{schema_name}' as schema;
USE {catalog_name};
SELECT * FROM {schema_name}.emp_details;
Recommendation
For minimal changes across all notebooks, I recommend Option 4 (Hybrid Approach):
1. Add just 2 lines to your param_notebook
2. Do a find-and-replace across SQL notebooks:
- Replace ${catalog.name} with {catalog_name}
- Replace ${schema.name} with {schema_name}