Random error: At least one column must be specified for the table?

[Jujiro]

I have the following code in a notebook. It is randomly giving me the error, "At least one column must be specified for the table." The error occurs (if at all it occurs) only on the first run after attaching to a cluster.

Cluster details:

Summary

5-10 Workers

320-640 GB Memory40-80 Cores1 Driver

64 GB Memory, 8 Cores Runtime

10.4.x-scala2.12 Apache Spark 3.2.1

 

Any ideas?

[Ajay-Pandey]

Create a support request databricks might help you in this issue.

[Jujiro]

The issue occurs randomly. The challenge is to recreate the issue for the support team to look. I am hoping that the folks who have experienced similar error, would comment, and then maybe the DBR folks would have something to investigate.

[Shan57]

Do you found any solution? Same error with me as well when i run below line.

spark.sql("DROP TABLE IF EXISTS tempdb.data_result");

[Jujiro]

Sorry, no solution yet.

[mathan_pillai]

I tried reproducing the issue in Databricks notebook, using 10.4 cluster and ran a few times. Unfortunately couldn't reproduce the issue. It runs successfully during each run. What is the frequency of this intermittent issue? If you re-run the command 10 times would it throw error once? Still would recommend to file a support ticket , so that we can take a deeper look at this.

[Jujiro]

So, basically, I have addressed the issue (for now) by putting the culprit statement in a try/catch block with a few retries. The error still occurs but clears in the second retry.

[mathan_pillai]

Are you having multiple threads that runs this statements concurrently ? If so the race condition could cause this issue, when trying to update the metastore.

[Jujiro]

I am not using any threading at all.

[mathan_pillai]

If you simply want to get rid of the table, you can drop the table using hive client, as well

https://learn.microsoft.com/en-us/azure/databricks/kb/metastore/drop-table-corruptedmetadata

[Jujiro]

I just wanted to simplify the code for illustration purposes. In my case the error occurs at the insert statement after the ALTER TABLE statement.

[Harold]

Please check if this could help or not:

spark.databricks.delta.catalog.update.enabled false