- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-09-2025 01:19 AM
Yes, it is possible for a Parquet file under a Delta table to consist of data written on different dates. This can happen because Delta Lake operations, such as inserts, updates, or deletes, create new Parquet files for the new records; along the same lines OPTIMIZE can compact multiple files into a single new file, which may contain data from different write operations. Therefore if a Parquet file contains data that is less than 5 years old, moving the file based on the file's creation date could indeed result in moving data that does not yet meet the 5-year criteria, so you would be moving data earlier than expected.
You can consider the following options, I've tried to sort them by complexity of implementation to keep things simpler:
- Use time-based partitioning (e.g., year or month) to align data movement with lifecycle policies.
- Use OPTIMIZE to manage file grouping and reduce data mixing.
- Run VACUUM to remove old, unreferenced files.
- Query Delta logs to identify and move only eligible data files. This requires scripting and a lot of testing, not straightforward.
- Exclude Delta folders from lifecycle policies or set conditions for movement. This adds a lot of manual management overhead, hence left it as last.