topic Re: How to refresh materialized view (Delta Live Table) without running a pipeline every time. in Get Started Discussions

How to refresh materialized view (Delta Live Table) without running a pipeline every time.

AbhiJ — Mon, 03 Jul 2023 11:24:05 GMT

Hello, My requirement is to created a materialized view (Live table) from a downstream table. The down stream table gets refreshed frequently and hence the materialized view needs to be recalculated. I don't want the pipe line to be in continuous mode (with streaming table's). My simple requirement is that the materilized view should get refreshed as per new data in the downstream table without having to run pipeline again (without any manual intervention).

Please let me know if anyone is having solution for it.

Re: How to refresh materialized view (Delta Live Table) without running a pipeline every time.

Chengcheng — Mon, 03 Jul 2023 11:38:57 GMT

If you want to update DLT asynchronously.(Without any manual intervention)

The only way is to separate it into two pipeline and schedule these two DLT pipeline separately.

Since DLT don't support any conditional run for now. You can't decide not to fresh a table when running it.

Re: How to refresh materialized view (Delta Live Table) without running a pipeline every time.

AbhiJ — Mon, 03 Jul 2023 11:53:31 GMT

Hi @Chengcheng

I am fine with manual refresh also and have tried "Refresh Table <TableName>" command but it doesn't refreshes the table with new data.

The Full refresh command "REFRESH TABLE <tableName> FULL; is giving below error inside notebook.

REFRESH statements cannot contain ' ', '\n', '\r', '\t' inside unquoted resource paths.(line 1, pos 0)

If any manual refresh is also available, I can run that sql command for that delta table while I am doing any updates to down stream table as of now.

Re: How to refresh materialized view (Delta Live Table) without running a pipeline every time.

Chengcheng — Tue, 04 Jul 2023 00:18:53 GMT

Hi AbhiJ,

You can use

Select tables for refresh

in your Delta Live Table UI, it allows you to select tables you would like to refresh.

And "Refresh Table <TableName>" may not work like you are thinking it is.

It's not used to refresh Delta Live Table, you can check it from the documentation in the following link.

https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-cache-refresh-table.html

But separate it into two pipeline is definitely better in your use case.