topic How can I activate enzyme for delta live tables (or dlt serverless) ? in Data Engineering

How can I activate enzyme for delta live tables (or dlt serverless) ?

nggianno — Fri, 16 Feb 2024 08:37:41 GMT

Hi!

I am using Delta Live Tables and especially materialized views and i want to run a dlt pipeline but not rerun the whole view that cost time, but rather only update and add only the values that have been changed. I saw that Enzyme and does this job and dlt serverless, but I do not have this choice in my dlt pipeline.

How can I activate that?

Re: How can I activate enzyme for delta live tables (or dlt serverless) ?

nggianno — Mon, 19 Feb 2024 10:14:43 GMT

Thank you for your answer.

What I meant with my question was the following: every time that I run a delta live table materialized view, I get a log of "COMPLETE RECOMPUTE" and I saw in a Databricks official video that there is an option to instead of "COMPLETE RECOMPUTE" to have "PARTITION OVERWRITE". How can I achieve that?

Re: How can I activate enzyme for delta live tables (or dlt serverless) ?

devmehta — Wed, 18 Sep 2024 12:36:20 GMT

I want to achieve the same as @nggianno. Please suggest a workaround

Re: How can I activate enzyme for delta live tables (or dlt serverless) ?

Mounika_Tarigop — Sat, 07 Dec 2024 05:27:40 GMT

To achieve "PARTITION OVERWRITE" instead of "COMPLETE RECOMPUTE" when running a Delta Live Table (DLT) materialized view, you need to configure your pipeline to use incremental refresh. This can be done by setting up your DLT pipeline to use serverless compute, which supports incremental refresh for materialized views.

https://docs.databricks.com/en/optimizations/incremental-refresh.html