topic Re: _sqldf bugs in GCP workspaces? in Data Engineering

_sqldf bugs in GCP workspaces?

jaredwolf — Thu, 27 Jul 2023 13:38:17 GMT

Utilizing GCP instances using the 12.2DBR ML runtime. Prior to ~7:10CT last night, _sqldf commands in notebooks to reference the previously executed %sql cell would work locally as well as in scheduled Workflow Job runs. Now it appears that the code will work locally with manual runs, but _sqldf commands now fail in the Workflow Job runs with a "NameError: name '_sqldf' is not defined".

Were any changes introduced last night that would impact this functionality?

Re: _sqldf bugs in GCP workspaces?

DannyPallotta — Fri, 28 Jul 2023 02:13:40 GMT

Hi Jaredwolf -

Just jumping in to say that we have noticed the same issue that you have described on Azure Databricks 12.2DBR (standard) in multiple regions.

In case it is of use to you, we have found updating to DBR 13+ seems to fix the issue.

But much like you I am interested to know what happened overnight, especially given it is an LTS runtime.

Re: _sqldf bugs in GCP workspaces?

Kayla — Tue, 22 Aug 2023 12:15:44 GMT

It looks like Azure was having the same issue, it might just be all 12.2 Photon clusters.
https://community.databricks.com/t5/data-engineering/sqldf-bugs-in-gcp-workspaces/td-p/38578
That post says it was been fixed, but last I checked it was still failing.
Fortunately for me, I don't use it much - I just converted my SQL cells to spark.sql("") in a Python cell.