Databricks run-time 12.2 LTS drop function problem

[sevvalmehder]

I am getting an error about the `drop function of pyspark` at a cluster using 12.2 LTS. When I check the error I see spark solved that bug, see SPARK-42444. Also when I check maintenance updates page, I saw this solved issue included the Databricks Runtime 12.2 at June 12. But I am still getting that error.

Do you have any idea?

[Sandeep]

Have you restarted the cluster after June 2? Also, can you try running it on DBR 13.x too?

[sevvalmehder]

Yeap, I restarted the cluster, and it run without getting an error after restarting. But it gets the error when I create a new cluster again.

Also, I tried 13.0 and everything goes well. The problem occurs only 12.x run-times.

[Anonymous]

Hi @Sevval Mehder​ 

Elevate our community by acknowledging exceptional contributions. Your participation in marking the best answers is a testament to our collective pursuit of knowledge.