Databricks Community

Jujiro · ‎12-02-2022

I have the following code in a notebook. It is randomly giving me the error, "At least one column must be specified for the table." The error occurs (if at all it occurs) only on the first run after attaching to a cluster.

Cluster details:

Summary

5-10 Workers

320-640 GB Memory40-80 Cores1 Driver

64 GB Memory, 8 Cores Runtime

10.4.x-scala2.12 Apache Spark 3.2.1

dbr-bug

Any ideas?

Ajay-Pandey · ‎12-07-2022

Create a support request databricks might help you in this issue.

Ajay Kumar Pandey

Jujiro · ‎12-07-2022

The issue occurs randomly. The challenge is to recreate the issue for the support team to look. I am hoping that the folks who have experienced similar error, would comment, and then maybe the DBR folks would have something to investigate.

Shan57 · ‎12-28-2022

Do you found any solution? Same error with me as well when i run below line.

spark.sql("DROP TABLE IF EXISTS tempdb.data_result");

Jujiro · ‎12-28-2022

Sorry, no solution yet.

mathan_pillai · ‎01-30-2023

I tried reproducing the issue in Databricks notebook, using 10.4 cluster and ran a few times. Unfortunately couldn't reproduce the issue. It runs successfully during each run. What is the frequency of this intermittent issue? If you re-run the command 10 times would it throw error once? Still would recommend to file a support ticket , so that we can take a deeper look at this.

Jujiro · ‎01-30-2023

So, basically, I have addressed the issue (for now) by putting the culprit statement in a try/catch block with a few retries. The error still occurs but clears in the second retry.

mathan_pillai · ‎01-30-2023

Are you having multiple threads that runs this statements concurrently ? If so the race condition could cause this issue, when trying to update the metastore.