How can I activate enzyme for delta live tables (or dlt serverless) ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2024 12:37 AM
Hi!
I am using Delta Live Tables and especially materialized views and i want to run a dlt pipeline but not rerun the whole view that cost time, but rather only update and add only the values that have been changed. I saw that Enzyme and does this job and dlt serverless, but I do not have this choice in my dlt pipeline.
How can I activate that?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-19-2024 02:14 AM
Thank you for your answer.
What I meant with my question was the following: every time that I run a delta live table materialized view, I get a log of "COMPLETE RECOMPUTE" and I saw in a Databricks official video that there is an option to instead of "COMPLETE RECOMPUTE" to have "PARTITION OVERWRITE". How can I achieve that?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-18-2024 05:36 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2024 09:27 PM
To achieve "PARTITION OVERWRITE" instead of "COMPLETE RECOMPUTE" when running a Delta Live Table (DLT) materialized view, you need to configure your pipeline to use incremental refresh. This can be done by setting up your DLT pipeline to use serverless compute, which supports incremental refresh for materialized views.
https://docs.databricks.com/en/optimizations/incremental-refresh.html
