Unable to migrate an empty parquet table to delta lake in Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-03-2024 10:08 AM
I'm trying to convert my Databricks Tables from Parquet to Delta. While most of the tables have data and are successfully converted to delta some of the empty parquet tables fail with an error message as below -
CONVERT TO DELTA <schema-name>.parquet_test_3
AnalysisException: Found no partition information in the catalog for table spark_catalog.<schema-name>.parquet_test_3. Have you run "MSCK REPAIR TABLE" on your table to discover partitions?
Has anyone tried converting empty parquet tables to delta format?
Note: MSCK REPAIR TABLE does not have any effect.
Inserting a dummy row into the table solves this issue but looking for any other best workarounds/solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-05-2024 05:37 AM
Thanks. Inserting a dummy row works but would it be addressed in a future release as a fix for COVERT TO DELTA function?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-05-2024 06:06 AM
Hello Bharathi,
Ideally the ETL job should not generate the empty parquet files in the respective location as it's an overhead to read the empty file and it's a not best practice.
Assuming this can be easily fix in ETL job while getting the rows count before generating the empty file.
