Databricks is expanding AUTO CDC to handle more of the difficult, real-world cases data engineers face, including bitemporal history, partial record updates, and open-source change data capture capabilities in Apache Spark.
Whatโs new
- Bitemporal history tracking: Bitemporal AUTO CDC tracks both business time, when a change was true in the real world, and system time, when the system learned about it. This makes it possible to reconstruct both what happened and what the system believed at a given point in time. Bitemporal AUTO CDC is currently in Beta.
- Partial Updates are generally available: When a source sends only the fields that changed, Partial Updates can prevent missing values from overwriting existing data. Teams can configure which columns should ignore incoming NULLs or use a source column to identify fields updated in each record.
- Less hand-written CDC logic: AUTO CDC provides declarative SQL and Python interfaces for applying changes, handling sequencing, and maintaining SCD Type 1 or Type 2 targets instead of relying on custom MERGE logic for every pipeline.
- Out-of-order changes are handled declaratively: Teams can define sequencing columns and let the pipeline process change events in the intended order, including events that arrive late or out of order.
- More open-source support: Databricks is contributing the Python API for AUTO CDC Type 1 to Apache Spark 4.2. The SQL interface has also been merged into Sparkโs main development branch for a future release.
Together, these updates extend AUTO CDC beyond basic current-state and history-tracking patterns. They help data teams build pipelines that are easier to maintain when records arrive late, updates are incomplete, or audit requirements require more than one view of time.
๐ Read the full post here
