lingareddy_Alva
Esteemed Contributor

Hi @sensanjoy 

Since ${param} syntax is deprecated in Databricks, here are the best approaches to make minimal changes across your SQL notebooks:
Option 1: Using Python Variables with f-strings (Recommended for minimal changes)
In your param_notebook:
# Set up widgets as before
dbutils.widgets.text("catalog", "catalog_de")
dbutils.widgets.text("schema", "emp")

# Create Python variables that can be used in SQL cells
catalog_name = dbutils.widgets.get("catalog")
schema_name = dbutils.widgets.get("schema")

# Optional: Set spark conf as well for backward compatibility
spark.conf.set("catalog.name", catalog_name)
spark.conf.set("schema.name", schema_name)

In your SQL notebook:
%run ../../config/param_notebook

-- Use Python variables in SQL cells
%sql
SELECT '{catalog_name}' as catalog, '{schema_name}' as schema;

USE {catalog_name};

SELECT * FROM {schema_name}.emp_details;

Option 2: Using spark.sql() with Python f-strings
In your SQL notebook:

%run ../../config/param_notebook

# Execute SQL with f-strings
spark.sql(f"USE {catalog_name}")

spark.sql(f"""
SELECT '{catalog_name}' as catalog, '{schema_name}' as schema
""").display()

spark.sql(f"""
SELECT * FROM {schema_name}.emp_details
""").display()

Option 3: Using Session Variables (Databricks SQL native approach)
In your param_notebook:

dbutils.widgets.text("catalog", "catalog_de")
dbutils.widgets.text("schema", "emp")

# Set session variables
spark.sql(f"SET VAR catalog_name = '{dbutils.widgets.get('catalog')}'")
spark.sql(f"SET VAR schema_name = '{dbutils.widgets.get('schema')}'")

In your SQL notebook:
%run ../../config/param_notebook

-- Use session variables
SELECT ${VAR.catalog_name} as catalog, ${VAR.schema_name} as schema;

USE IDENTIFIER(${VAR.catalog_name});

SELECT * FROM IDENTIFIER(CONCAT(${VAR.schema_name}, '.emp_details'));

Option 4: Hybrid Approach (Best for your current setup)
Keep your param_notebook mostly unchanged:

dbutils.widgets.text("catalog", "catalog_de")
spark.conf.set("catalog.name", dbutils.widgets.get("catalog"))

dbutils.widgets.text("schema", "emp")
spark.conf.set("schema.name", dbutils.widgets.get("schema"))

# Add these lines for the new approach
catalog_name = dbutils.widgets.get("catalog")
schema_name = dbutils.widgets.get("schema")

Update SQL notebooks minimally:
%run ../../config/param_notebook

-- Replace ${catalog.name} with {catalog_name}
SELECT '{catalog_name}' as catalog, '{schema_name}' as schema;

USE {catalog_name};

SELECT * FROM {schema_name}.emp_details;


Recommendation
For minimal changes across all notebooks, I recommend Option 4 (Hybrid Approach):
1. Add just 2 lines to your param_notebook
2. Do a find-and-replace across SQL notebooks:
- Replace ${catalog.name} with {catalog_name}
- Replace ${schema.name} with {schema_name}

 

LR