How can I activate enzyme for delta live tables (or dlt serverless) ?

[nggianno]

Hi!

I am using Delta Live Tables and especially materialized views and i want to run a dlt pipeline but not rerun the whole view that cost time, but rather only update and add only the values that have been changed. I saw that Enzyme and does this job and dlt serverless, but I do not have this choice in my dlt pipeline. 

How can I activate that?

[Kaniz]

Hi @nggianno, Delta Live Tables (DLT) provides a powerful way to manage data pipelines and materialized views. When it comes to updating materialized views without rerunning the entire pipeline, there are a few considerations:

1. Automatic Monitoring and Updates:

   • Both materialized views and streaming tables in DLT can be updated in either execution mode.
   • To avoid unnecessary processing in continuous execution mode, DLT pipelines automatically monitor dependent Delta tables.
   • They perform an update only when the contents of those dependent tables have changed¹².

2. Separate Pipelines:

   • If you want to update DLT asynchronously (without manual intervention), consider separating the process into two pipelines:
     • Pipeline A: Refresh the downstream table (which gets updated frequently).
     • Pipeline B: Create or update the materialized view based on the refreshed downstream table.
   • Schedule these two DLT pipelines separately.
   • Unfortunately, DLT does not currently support conditional runs, so you cannot decide not to refresh ...³.

3. Manual Refresh:

   • While DLT doesn’t directly support incremental updates for materialized views, you can manually refresh a Delta table using SQL commands.
   • The REFRESH TABLE <TableName> command is used to refresh Delta tables, not specifically materialized views.
   • If you need to refresh a materialized view, consider using the SELECT TABLES FOR REFRESH option in your Delta Live Table UI.
   • This allows you to select specific tables you want to refresh³.

In summary, consider separating your pipeline into two parts and schedule them independently. While DLT doesn’t directly provide incremental updates for materialized views, you can achieve similar results by carefully managing your pipelines and using manual refresh options when needed. 🚀

[nggianno]

Thank you for your answer.

What I meant with my question was the following: every time that I run a delta live table materialized view, I get a log of "COMPLETE RECOMPUTE" and I saw in a Databricks official video that there is an option to instead of "COMPLETE RECOMPUTE" to have "PARTITION OVERWRITE". How can I achieve that?