Auto SCD API Tombstone Garbage Collection
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yesterday
Are there any settings that can be used to influence the frequency at which the auto SCD API runs the tombstone garbage collection process in Spark Declarative Pipelines? I've seen a couple community posts that referenced the following:
pipelines.applyChanges.tombstoneGCFrequencyInSeconds
pipelines.cdc.tombstoneGCFrequencyInSeconds
But these are not covered in any of the official Databricks docs am having trouble getting either of these to work at all. When you sequence by a struct, the GC deletes become very inefficient from a data skipping perspective, so hoping there is a way to get these to run less frequently