Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2021 02:13 AM
the auto optimize is sufficient, unless you run into performance issues.
Then I would trigger an optimize. This will generate files of 1GB (so larger than the standard size of auto optimize). And of course the Z-Order if necessary.
The suggestion to run optimize will probably be a proposal to apply Z-ordering because you use a highly selective filter in your notebook.
Z-ordering is a very interesting optimization technique but one should check what the best ordering could be. So depending on the case this can be interesting or not.
Auto-optimize does not apply z-ordering.
https://docs.microsoft.com/en-us/azure/databricks/delta/optimizations/auto-optimize