Historical Data Clean-up from Silver tables
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-18-2024 12:38 AM
Hi Everyone,
I need your help/suggestion.
We are using a DLT framework for our ELT process, data is received from the Source to the RAW layer in parquet format. This raw data is loaded to the Bronze layer which acts like a history table. From the BRONZE layer data is loaded to Silver using Merge login(upsert) in Delta format.
As part of cost savings, we cleaned up unwanted/old data from Silver tables(records older than 2017)
we used the delete query to delete these records and we ran VACCUME after clean-up to claim the free space.
we saw very good space savings after this activity.
But a few weeks later we observed above silver table's ADLS space was the same as before the cleanup
But when I query the table I don't see the deleted records.
Can someone please help me what is the reason for the space increase and how to get free space permanently after the cleanup?