Databricks run-time 12.2 LTS drop function problem
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2023 05:09 AM
I am getting an error about the `drop function of pyspark` at a cluster using 12.2 LTS. When I check the error I see spark solved that bug, see SPARK-42444. Also when I check maintenance updates page, I saw this solved issue included the Databricks Runtime 12.2 at June 12. But I am still getting that error.
Do you have any idea?
- Labels:
-
Apache spark
-
Databricks Runtime
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2023 05:48 AM
Have you restarted the cluster after June 2? Also, can you try running it on DBR 13.x too?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2023 03:28 PM
Yeap, I restarted the cluster, and it run without getting an error after restarting. But it gets the error when I create a new cluster again.
Also, I tried 13.0 and everything goes well. The problem occurs only 12.x run-times.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2023 10:27 PM
Hi @Sevval Mehder
Elevate our community by acknowledging exceptional contributions. Your participation in marking the best answers is a testament to our collective pursuit of knowledge.