01-03-2024 10:08 AM
I'm trying to convert my Databricks Tables from Parquet to Delta. While most of the tables have data and are successfully converted to delta some of the empty parquet tables fail with an error message as below -
CONVERT TO DELTA <schema-name>.parquet_test_3
AnalysisException: Found no partition information in the catalog for table spark_catalog.<schema-name>.parquet_test_3. Have you run "MSCK REPAIR TABLE" on your table to discover partitions?
Has anyone tried converting empty parquet tables to delta format?
Note: MSCK REPAIR TABLE does not have any effect.
Inserting a dummy row into the table solves this issue but looking for any other best workarounds/solution.
01-04-2024 12:45 AM
Hi @Bharathi-Rajen, you could write a script that checks if a Parquet table is empty before converting it to Delta. If the table is empty, the script could insert a dummy row, perform the conversion, and then remove the dummy row from the Delta table.
01-05-2024 05:37 AM
Thanks. Inserting a dummy row works but would it be addressed in a future release as a fix for COVERT TO DELTA function?
01-05-2024 06:06 AM
Hello Bharathi,
Ideally the ETL job should not generate the empty parquet files in the respective location as it's an overhead to read the empty file and it's a not best practice.
Assuming this can be easily fix in ETL job while getting the rows count before generating the empty file.
01-18-2024 01:32 AM
Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?
This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group