Liquid clustering with incremental ingestion
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2024 04:58 AM
We ingest data incrementally from a database into delta tables using a column updatedUtc. This column is a datetime and is updated when the row in the database table changes. What about using this non-mutable column in "cluster by"? Would it require executing OPTIMIZE more often?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2024 06:32 AM
It recommended to run optimize query in scheduled manner
https://docs.databricks.com/en/delta/clustering.html#how-to-trigger-clustering
