topic Random error: At least one column must be specified for the table? in Data Engineering

Random error: At least one column must be specified for the table?

Jujiro — Fri, 02 Dec 2022 17:25:59 GMT

I have the following code in a notebook. It is randomly giving me the error, "At least one column must be specified for the table." The error occurs (if at all it occurs) only on the first run after attaching to a cluster.

Cluster details:

Summary

5-10 Workers

320-640 GB Memory40-80 Cores1 Driver

64 GB Memory, 8 Cores Runtime

10.4.x-scala2.12 Apache Spark 3.2.1

Any ideas?

Re: Random error: At least one column must be specified for the table?

Ajay-Pandey — Wed, 07 Dec 2022 09:03:17 GMT

Create a support request databricks might help you in this issue.

Re: Random error: At least one column must be specified for the table?

Jujiro — Wed, 07 Dec 2022 16:39:36 GMT

The issue occurs randomly. The challenge is to recreate the issue for the support team to look. I am hoping that the folks who have experienced similar error, would comment, and then maybe the DBR folks would have something to investigate.

Re: Random error: At least one column must be specified for the table?

Shan57 — Wed, 28 Dec 2022 12:20:12 GMT

Do you found any solution? Same error with me as well when i run below line.

spark.sql("DROP TABLE IF EXISTS tempdb.data_result");

Re: Random error: At least one column must be specified for the table?

Jujiro — Wed, 28 Dec 2022 14:09:37 GMT

Sorry, no solution yet.

Re: Random error: At least one column must be specified for the table?

mathan_pillai — Mon, 30 Jan 2023 17:53:24 GMT

I tried reproducing the issue in Databricks notebook, using 10.4 cluster and ran a few times. Unfortunately couldn't reproduce the issue. It runs successfully during each run. What is the frequency of this intermittent issue? If you re-run the command 10 times would it throw error once? Still would recommend to file a support ticket , so that we can take a deeper look at this.

Re: Random error: At least one column must be specified for the table?

mathan_pillai — Mon, 30 Jan 2023 18:11:52 GMT

Are you having multiple threads that runs this statements concurrently ? If so the race condition could cause this issue, when trying to update the metastore.

Re: Random error: At least one column must be specified for the table?

mathan_pillai — Mon, 30 Jan 2023 18:19:35 GMT

If you simply want to get rid of the table, you can drop the table using hive client, as well

https://learn.microsoft.com/en-us/azure/databricks/kb/metastore/drop-table-corruptedmetadata

Re: Random error: At least one column must be specified for the table?

Jujiro — Mon, 30 Jan 2023 18:28:58 GMT

So, basically, I have addressed the issue (for now) by putting the culprit statement in a try/catch block with a few retries. The error still occurs but clears in the second retry.

Re: Random error: At least one column must be specified for the table?

Jujiro — Mon, 30 Jan 2023 18:29:49 GMT

I am not using any threading at all.

Re: Random error: At least one column must be specified for the table?

Jujiro — Mon, 30 Jan 2023 18:31:28 GMT

I just wanted to simplify the code for illustration purposes. In my case the error occurs at the insert statement after the ALTER TABLE statement.

Re: Random error: At least one column must be specified for the table?

Harold — Tue, 06 Jun 2023 01:21:51 GMT

Please check if this could help or not:

spark.databricks.delta.catalog.update.enabled false