cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Materialized views Refresh strategy in Triggerd vs Continuous DLT mode

ManojReddy
New Contributor II

Does Materialized views gets completely recalculated when we trigger DLT pipeline? Can't we start from where it left?
In continuous mode of DLT pipeline Materialized view tries to optimizes the updates and computes the data?

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @ManojReddy , Certainly! Let’s break it down in a way that’s easy to understand:

 

Materialized Views:

  • Imagine materialized views as precomputed results. They store data that you can query efficiently.
  • When you refresh a materialized view, it recalculates the results based on changes in the original data.
  • Think of it like a snapshot that updates itself whenever the underlying data changes.

DLT Pipelines:

  • DLT pipelines are like powerful data transformers. They process data, apply logic, and create the desired output.
  • When you trigger a DLT pipeline, it processes data through query rules and transformations.
  • For materialized views, the pipeline ensures that the results are always accurate and up-to-date.

Choosing the Right Dataset Type:

  • If you have complex queries, use views. They don’t store results but help break down big queries.
  • For shared data across multiple queries, go for materialized views. They recompute results every time you query them.
  • If you’re dealing with constantly growing data, consider streaming tables. They compute results incrementally.

In a nutshell, materialized views keep things accurate, DLT pipelines do the heavy lifting, and choosing the right dataset type ensures efficiency. 🌟

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group