cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Random error: At least one column must be specified for the table?

Jujiro
New Contributor III

I have the following code in a notebook. It is randomly giving me the error, "At least one column must be specified for the table." The error occurs (if at all it occurs) only on the first run after attaching to a cluster.

Cluster details:

Summary

5-10 Workers

320-640 GB Memory40-80 Cores1 Driver

64 GB Memory, 8 Cores Runtime

10.4.x-scala2.12 Apache Spark 3.2.1

dbr-bug 

Any ideas?

11 REPLIES 11

Ajay-Pandey
Esteemed Contributor III

Create a support request databricks might help you in this issue.

Jujiro
New Contributor III

The issue occurs randomly. The challenge is to recreate the issue for the support team to look. I am hoping that the folks who have experienced similar error, would comment, and then maybe the DBR folks would have something to investigate.

Shan57
New Contributor II

image.pngDo you found any solution? Same error with me as well when i run below line.

spark.sql("DROP TABLE IF EXISTS tempdb.data_result");

Jujiro
New Contributor III

Sorry, no solution yet.

mathan_pillai
Valued Contributor
Valued Contributor

I tried reproducing the issue in Databricks notebook, using 10.4 cluster and ran a few times. Unfortunately couldn't reproduce the issue. It runs successfully during each run. What is the frequency of this intermittent issue? If you re-run the command 10 times would it throw error once? Still would recommend to file a support ticket , so that we can take a deeper look at this.

Jujiro
New Contributor III

So, basically, I have addressed the issue (for now) by putting the culprit statement in a try/catch block with a few retries. The error still occurs but clears in the second retry.

mathan_pillai
Valued Contributor
Valued Contributor

Are you having multiple threads that runs this statements concurrently ? If so the race condition could cause this issue, when trying to update the metastore.

Jujiro
New Contributor III

I am not using any threading at all.

mathan_pillai
Valued Contributor
Valued Contributor

If you simply want to get rid of the table, you can drop the table using hive client, as well

https://learn.microsoft.com/en-us/azure/databricks/kb/metastore/drop-table-corruptedmetadata

Jujiro
New Contributor III

I just wanted to simplify the code for illustration purposes. In my case the error occurs at the insert statement after the ALTER TABLE statement.

Harold
New Contributor II

Please check if this could help or not:

spark.databricks.delta.catalog.update.enabled false

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!