topic Re: Unable to migrate an empty parquet table to delta lake in Databricks in Get Started Discussions

Unable to migrate an empty parquet table to delta lake in Databricks

Bharathi-Rajen — Wed, 03 Jan 2024 18:08:18 GMT

I'm trying to convert my Databricks Tables from Parquet to Delta. While most of the tables have data and are successfully converted to delta some of the empty parquet tables fail with an error message as below -

CONVERT TO DELTA <schema-name>.parquet_test_3

AnalysisException: Found no partition information in the catalog for table spark_catalog.<schema-name>.parquet_test_3. Have you run "MSCK REPAIR TABLE" on your table to discover partitions?

Has anyone tried converting empty parquet tables to delta format?

Note: MSCK REPAIR TABLE does not have any effect.

Inserting a dummy row into the table solves this issue but looking for any other best workarounds/solution.

Re: Unable to migrate an empty parquet table to delta lake in Databricks

Bharathi-Rajen — Fri, 05 Jan 2024 13:37:25 GMT

Thanks. Inserting a dummy row works but would it be addressed in a future release as a fix for COVERT TO DELTA function?

Re: Unable to migrate an empty parquet table to delta lake in Databricks

BR_DatabricksAI — Fri, 05 Jan 2024 14:06:08 GMT

Hello Bharathi,

Ideally the ETL job should not generate the empty parquet files in the respective location as it's an overhead to read the empty file and it's a not best practice.

Assuming this can be easily fix in ETL job while getting the rows count before generating the empty file.