Optimize operation with big increase in numRemovedFiles/numRemovedBytes/numAddedFiles/numAddedBytes
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2023 01:47 PM
Hello,
I have a daily loading process for a delta table and has a ‘optimize table’ step at the end. The optimize operation used to take about 5 minutes, but now takes about 3.5 hours. One thing I noticed from 'describe history' is the operationMetrics captured some increase in numRemovedFiles/numRemovedBytes/numAddedFiles/numAddedBytes (highlighted in red below).
I checked the source files, which are of similar size as before. I am wondering where I should look to see what caused the increase in numRemovedFiles/numRemovedBytes/numAddedFiles/numAddedBytes?
Thank you.
Labels:
- Labels:
-
Optimize Command