Converting data that is in Delta format to plain parquet format
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2021 02:36 PM
Many a times there is a need to convert Delta tables from Delta format to plain parquet format for a number of reasons, what is the best way to do that?
Labels:
- Labels:
-
Delta Format
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2021 06:11 PM
You can easily convert a Delta table back to a Parquet table using the following steps:
- If you have performed Delta Lake operations that can change the data files (for example, delete or merge, run vacuum with retention of 0 hours to delete all data files that do not belong to the latest version of the table.
- Delete the _delta_log directory in the table directory.
Refrence and mir info - https://docs.databricks.com/delta/delta-utility.html#convert-a-delta-table-to-a-parquet-table