cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to migrate an empty parquet table to delta lake in Databricks

Bharathi-Rajen
New Contributor II

I'm trying to convert my Databricks Tables from Parquet to Delta. While most of the tables have data and are successfully converted to delta some of the empty parquet tables fail with an error message as below -

CONVERT TO DELTA <schema-name>.parquet_test_3

AnalysisException: Found no partition information in the catalog for table spark_catalog.<schema-name>.parquet_test_3. Have you run "MSCK REPAIR TABLE" on your table to discover partitions?

Has anyone tried converting empty parquet tables to delta format?

Note: MSCK REPAIR TABLE does not have any effect.

Inserting a dummy row into the table solves this issue but looking for any other best workarounds/solution.

2 REPLIES 2

Bharathi-Rajen
New Contributor II

Thanks. Inserting a dummy row works but would it be addressed in a future release as a fix for COVERT TO DELTA function?

BR_DatabricksAI
Contributor

Hello Bharathi, 

Ideally the ETL job should not generate the empty parquet files in the respective location as it's an overhead to read the empty file and it's a not best practice.

Assuming this can be easily fix in ETL job while getting the rows count before generating the empty file.