Unable to migrate an empty parquet table to delta lake in Databricks

[Bharathi-Rajen]

I'm trying to convert my Databricks Tables from Parquet to Delta. While most of the tables have data and are successfully converted to delta some of the empty parquet tables fail with an error message as below -

CONVERT TO DELTA <schema-name>.parquet_test_3

AnalysisException: Found no partition information in the catalog for table spark_catalog.<schema-name>.parquet_test_3. Have you run "MSCK REPAIR TABLE" on your table to discover partitions?

Has anyone tried converting empty parquet tables to delta format?

Note: MSCK REPAIR TABLE does not have any effect.

Inserting a dummy row into the table solves this issue but looking for any other best workarounds/solution.

[Kaniz_Fatma]

Hi @Bharathi-Rajen, you could write a script that checks if a Parquet table is empty before converting it to Delta. If the table is empty, the script could insert a dummy row, perform the conversion, and then remove the dummy row from the Delta table.

[Bharathi-Rajen]

Thanks. Inserting a dummy row works but would it be addressed in a future release as a fix for COVERT TO DELTA function?

[BR_DatabricksAI]

Hello Bharathi, 

Ideally the ETL job should not generate the empty parquet files in the respective location as it's an overhead to read the empty file and it's a not best practice.

Assuming this can be easily fix in ETL job while getting the rows count before generating the empty file. 

[Kaniz_Fatma]

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!