cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

_sqldf bugs in GCP workspaces?

jaredwolf
New Contributor II

Utilizing GCP instances using the 12.2DBR ML runtime. Prior to ~7:10CT last night, _sqldf commands in notebooks to reference the previously executed %sql cell would work locally as well as in scheduled Workflow Job runs. Now it appears that the code will work locally with manual runs, but  _sqldf commands now fail in the Workflow Job runs with a "NameError: name '_sqldf' is not defined".

Were any changes introduced last night that would impact this functionality?

2 REPLIES 2

DannyPallotta
New Contributor III

Hi Jaredwolf -

Just jumping in to say that we have noticed the same issue that you have described on Azure Databricks 12.2DBR (standard) in multiple regions.

In case it is of use to you, we have found updating to DBR 13+ seems to fix the issue.

But much like you I am interested to know what happened overnight, especially given it is an LTS runtime.

Kayla
Valued Contributor

It looks like Azure was having the same issue, it might just be all 12.2 Photon clusters.
https://community.databricks.com/t5/data-engineering/sqldf-bugs-in-gcp-workspaces/td-p/38578 
That post says it was been fixed, but last I checked it was still failing. 
Fortunately for me, I don't use it much - I just converted my SQL cells to spark.sql("") in a Python cell.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group