topic Re: Databricks run-time 12.2 LTS drop function problem in Data Engineering

Databricks run-time 12.2 LTS drop function problem

sevvalmehder — Wed, 14 Jun 2023 12:09:04 GMT

I am getting an error about the `drop function of pyspark` at a cluster using 12.2 LTS. When I check the error I see spark solved that bug, see SPARK-42444. Also when I check maintenance updates page, I saw this solved issue included the Databricks Runtime 12.2 at June 12. But I am still getting that error.

Do you have any idea?

Re: Databricks run-time 12.2 LTS drop function problem

Sandeep — Wed, 14 Jun 2023 12:48:16 GMT

Have you restarted the cluster after June 2? Also, can you try running it on DBR 13.x too?

Re: Databricks run-time 12.2 LTS drop function problem

sevvalmehder — Wed, 14 Jun 2023 22:28:35 GMT

Yeap, I restarted the cluster, and it run without getting an error after restarting. But it gets the error when I create a new cluster again.

Also, I tried 13.0 and everything goes well. The problem occurs only 12.x run-times.

Re: Databricks run-time 12.2 LTS drop function problem

Anonymous — Thu, 15 Jun 2023 05:27:21 GMT

Hi @Sevval Mehder

Elevate our community by acknowledging exceptional contributions. Your participation in marking the best answers is a testament to our collective pursuit of knowledge.