cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to refresh materialized view (Delta Live Table) without running a pipeline every time.

AbhiJ
New Contributor III

Hello, My requirement is to created a materialized view (Live table) from a downstream table. The down stream table gets refreshed frequently and hence the materialized view needs to be recalculated. I don't want the pipe line to be in continuous mode (with streaming table's). My simple requirement is that the materilized view should get refreshed as per new data in the downstream table without having to run pipeline again (without any manual intervention).

Please let me know if anyone is having solution for it.

3 REPLIES 3

Chengcheng
New Contributor III

If you want to update DLT asynchronously.(Without any manual intervention)

The only way is to separate it into two pipeline and schedule these two DLT pipeline separately.

Since DLT don't support any conditional run for now. You can't decide not to fresh a table when running it.

AbhiJ
New Contributor III

Hi @Chengcheng 

I am fine with manual refresh also and have tried "Refresh Table <TableName>" command but it doesn't refreshes the table with new data.

The Full refresh command "REFRESH TABLE <tableName> FULL; is giving below error inside notebook.

REFRESH statements cannot contain ' ', '\n', '\r', '\t' inside unquoted resource paths.(line 1, pos 0)

If any  manual refresh is also available, I can run that sql command for that delta table while I am doing any updates to down stream table as of now.

 

Chengcheng
New Contributor III

Hi AbhiJ

You can use 

  • Select tables for refresh

in your Delta Live Table UI, it allows you to select tables you would like to refresh. 

 

And "Refresh Table <TableName>" may not work like you are thinking it is.

It's not used to refresh Delta Live Table, you can check it from the documentation in the following link.

https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-cache-refresh-table.html

 

But separate it into two pipeline is definitely better in your use case.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group