Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-14-2025 06:39 AM
Databricks Delta Live Tables (DLT) supports liquid clustering for both streaming tables and materialized views (MVs), not just streaming tables. This means liquid clustering is available for tables managed through DLT in both Preview and Current channels, including materialized views created via DLT pipelines
As for the merge statement.
Merge does work with Delta Live Tables (DLT) pipelines but not as a direct statement; instead, the functionality is provided via the
APPLY CHANGES INTO operation. This operation serves as the equivalent of the MERGE INTO command for Delta Lake tables, enabling users to process updates, inserts, and deletes from source tables.Key details about
APPLY CHANGES INTO in DLT pipelines:1. Supported Operations: It applies
INSERT and UPDATE events from the source dataset by matching primary keys and event sequencing to maintain data consistency. DELETE operations can also be handled using statements like APPLY AS DELETE WHEN in SQL, or its Python equivalent.2. Pipeline Compatibility: The target table for
APPLY CHANGES INTO must be a live table and cannot be a streaming live table.3. Configuration Requirement: The operation needs to be explicitly enabled in the pipeline settings by adding and enabling the
applyChanges configuration.This approach bypasses limitations of simple streaming table operations, which are restricted to append-only queries, making it suitable for handling use cases requiring incremental updates to tables.
Hope this helps, Lou.
