topic Re: _sqldf not defined on Azure job cluster v12.2 in Data Engineering

_sqldf not defined on Azure job cluster v12.2

piterpan — Fri, 28 Jul 2023 07:53:09 GMT

Since yesterday we have errors in notebooks that were previously working.

NameError: name '_sqldf' is not defined

It was working previously.

We are on Azure databricks, usng job pool
Driver: Standard_D4s_v5 · Workers: Standard_D4s_v5 · 1-6 workers · from pool · 12.2 LTS

We have opened a ticket on Azure, no response for now 😕

Re: _sqldf not defined on Azure job cluster v12.2

Kale-S — Fri, 28 Jul 2023 12:41:37 GMT

I have the same error since yesterday.

I have rewrote all my magic SQL (%SQL) cells with the spark function:
query = """
select *
from table
"""

df = spark.sql(query)

Re: _sqldf not defined on Azure job cluster v12.2

samirsaliba — Fri, 28 Jul 2023 12:45:24 GMT

Same here, also on Azure Databricks. Cluster version is 12.2 LTS Photon.

Re: _sqldf not defined on Azure job cluster v12.2

ViniciusTavares — Fri, 28 Jul 2023 12:57:21 GMT

Today I had the same error, the df _sqldf stopped working

Re: _sqldf not defined on Azure job cluster v12.2

aholts — Fri, 28 Jul 2023 13:26:36 GMT

Having same issue since Thursday

Re: _sqldf not defined on Azure job cluster v12.2

whitebulb — Tue, 01 Aug 2023 03:35:57 GMT

Same here. It was working completely fine a couple weeks ago.

Re: _sqldf not defined on Azure job cluster v12.2

Tharun-Kumar — Tue, 01 Aug 2023 04:37:57 GMT

@piterpan

This was a regression issue which impacted the jobs where _sqldf was referenced and the notebook those weren't run interactively. Our Engineering team has fixed this issue yesterday.

Could you check whether you are still facing the issue?

Re: _sqldf not defined on Azure job cluster v12.2

piterpan — Thu, 03 Aug 2023 10:18:34 GMT

Thank for your feedback.
We noticed it's working on v13.2 and we have been using v13.2 as a workaround.
I am waiting for the test on v12.2 and I'll confirm you if it's resolved for us.

Re: _sqldf not defined on Azure job cluster v12.2

piterpan — Thu, 03 Aug 2023 17:05:53 GMT

I can confirm it's working again.