How to refresh materialized view (Delta Live Table) without running a pipeline every time.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2023 04:22 AM - edited 07-03-2023 04:24 AM
Hello, My requirement is to created a materialized view (Live table) from a downstream table. The down stream table gets refreshed frequently and hence the materialized view needs to be recalculated. I don't want the pipe line to be in continuous mode (with streaming table's). My simple requirement is that the materilized view should get refreshed as per new data in the downstream table without having to run pipeline again (without any manual intervention).
Please let me know if anyone is having solution for it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2023 04:38 AM
If you want to update DLT asynchronously.(Without any manual intervention)
The only way is to separate it into two pipeline and schedule these two DLT pipeline separately.
Since DLT don't support any conditional run for now. You can't decide not to fresh a table when running it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2023 04:53 AM
Hi @Chengcheng
I am fine with manual refresh also and have tried "Refresh Table <TableName>" command but it doesn't refreshes the table with new data.
The Full refresh command "REFRESH TABLE <tableName> FULL; is giving below error inside notebook.
REFRESH statements cannot contain ' ', '\n', '\r', '\t' inside unquoted resource paths.(line 1, pos 0)
If any manual refresh is also available, I can run that sql command for that delta table while I am doing any updates to down stream table as of now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2023 05:18 PM
Hi AbhiJ,
You can use
- Select tables for refresh
in your Delta Live Table UI, it allows you to select tables you would like to refresh.
And "Refresh Table <TableName>" may not work like you are thinking it is.
It's not used to refresh Delta Live Table, you can check it from the documentation in the following link.
https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-cache-refresh-table.html
But separate it into two pipeline is definitely better in your use case.