Hi @Somia,

If you run that function in all-purpose does it work fine?

Is there any equivalent to _sqldf for SQL Warehouses?

I switched my notebook back to all purpose compute as _sqldf is not supported with sql warehouse notebook. It didn't work as explained above. 

@Somia 

To replicate your scenario, you need an All-Purpose Cluster and a notebook defaulted to python language. Then query a table using %sql as below. This creates a temp dataframe for you to use it in the python cells. Keep in mind, that this dataframe keep changing as you execute a different %sql cell.

%sql
-- cell 1
select * from catalog.schema.123_sample

# cell 2
display(_sqldf)

To summarize, the %sql magic command behaves differently depending on whether your Databricks notebook is connected to an All-Purpose cluster or a SQL Warehouse.

1. All-Purpose Cluster: %sql creates a DataFrame named _sqldf that you can use in subsequent Python cells.
2. SQL Warehouse: %sql executes the query but does not create the _sqldf DataFrame.

Please let me know for anything, else mark it as a solution.

@Somia _sqldf is a pyspark df not a sql object.

It works only for such direction 

exec sql -> call _sqldf in pyspark

@Somia Can you mark my detailed explanation as solution that helped to resolve your issue.