โ07-03-2023 04:22 AM - edited โ07-03-2023 04:24 AM
Hello, My requirement is to created a materialized view (Live table) from a downstream table. The down stream table gets refreshed frequently and hence the materialized view needs to be recalculated. I don't want the pipe line to be in continuous mode (with streaming table's). My simple requirement is that the materilized view should get refreshed as per new data in the downstream table without having to run pipeline again (without any manual intervention).
Please let me know if anyone is having solution for it.
โ07-03-2023 04:38 AM
If you want to update DLT asynchronously.(Without any manual intervention)
The only way is to separate it into two pipeline and schedule these two DLT pipeline separately.
Since DLT don't support any conditional run for now. You can't decide not to fresh a table when running it.
โ07-03-2023 04:53 AM
Hi @Chengcheng
I am fine with manual refresh also and have tried "Refresh Table <TableName>" command but it doesn't refreshes the table with new data.
The Full refresh command "REFRESH TABLE <tableName> FULL; is giving below error inside notebook.
REFRESH statements cannot contain ' ', '\n', '\r', '\t' inside unquoted resource paths.(line 1, pos 0)
If any manual refresh is also available, I can run that sql command for that delta table while I am doing any updates to down stream table as of now.
โ07-03-2023 05:18 PM
Hi AbhiJ,
You can use
in your Delta Live Table UI, it allows you to select tables you would like to refresh.
And "Refresh Table <TableName>" may not work like you are thinking it is.
It's not used to refresh Delta Live Table, you can check it from the documentation in the following link.
https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-cache-refresh-table.html
But separate it into two pipeline is definitely better in your use case.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group