cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Unable to migrate an empty parquet table to delta lake in Databricks

Bharathi-Rajen
New Contributor II

I'm trying to convert my Databricks Tables from Parquet to Delta. While most of the tables have data and are successfully converted to delta some of the empty parquet tables fail with an error message as below -

CONVERT TO DELTA <schema-name>.parquet_test_3

AnalysisException: Found no partition information in the catalog for table spark_catalog.<schema-name>.parquet_test_3. Have you run "MSCK REPAIR TABLE" on your table to discover partitions?

Has anyone tried converting empty parquet tables to delta format?

Note: MSCK REPAIR TABLE does not have any effect.

Inserting a dummy row into the table solves this issue but looking for any other best workarounds/solution.

2 REPLIES 2

Bharathi-Rajen
New Contributor II

Thanks. Inserting a dummy row works but would it be addressed in a future release as a fix for COVERT TO DELTA function?

BR_DatabricksAI
Contributor

Hello Bharathi, 

Ideally the ETL job should not generate the empty parquet files in the respective location as it's an overhead to read the empty file and it's a not best practice.

Assuming this can be easily fix in ETL job while getting the rows count before generating the empty file. 

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group