As data platforms move toward more declarative and incremental processing, one question comes up frequently when building pipelines in Databricks:
Should I use a Materialized View or a Streaming Table?
Both are important building blocks in Lakeflow pipelines, but they solve different problems.
Materialized Views
A Materialized View stores the precomputed results of a query rather than calculating the result every time someone queries it.
When the upstream data changes, the Materialized View can be refreshed so that the stored result reflects the latest data.
This makes Materialized Views especially useful for transformations such as:
Aggregations
Data cleaning
Joins
Business level transformations
BI reporting tables
Dashboard datasets
Databricks can also incrementally refresh Materialized Views when the query and underlying data support it, which can avoid recomputing the entire dataset. (Databricks Documentation)
A simple example:
CREATE OR REFRESH MATERIALIZED VIEW daily_sales
AS
SELECT
order_date,
SUM(amount) AS total_sales
FROM sales
GROUP BY order_date;
Instead of recalculating the aggregation every time a user queries daily_sales, Databricks maintains the result for us.
Materialized Views can also be refreshed manually, on a schedule, or automatically based on upstream updates. (Databricks Documentation)
Streaming Tables
Streaming Tables are designed for data that continuously arrives.
Typical examples include:
Application events
IoT data
Kafka or Event Hub streams
Cloud file ingestion
Transaction events
Operational logs
In a Lakeflow pipeline, a Streaming Table can be created with SQL:
CREATE OR REFRESH STREAMING TABLE bronze_orders
AS
SELECT *
FROM STREAM read_files(
'/Volumes/catalog/schema/raw/orders',
format => 'json'
);
The important difference is that the source is processed using streaming semantics.
Instead of processing the complete dataset during every pipeline update, the pipeline processes newly arriving data incrementally.
Databricks uses the STREAM keyword when a source should be read using streaming semantics. (Databricks Documentation)
A Simple Architecture
A common architecture can combine both concepts.
Source Data
โ
Streaming Table
โ
Bronze
โ
Streaming / Transformation Layer
โ
Silver
โ
Materialized View
โ
Gold / BI / Analytics
For example, incoming orders can first land in a Streaming Table.
The data can then be cleaned and validated in the Silver layer.
Finally, a Materialized View can aggregate the data for dashboards, reporting, or analytics.
This allows streaming ingestion and analytical transformations to work together instead of treating batch and streaming as completely separate architectures.
When Should You Use Each One?
Use a Streaming Table when the main requirement is continuously processing newly arriving data.
Use a Materialized View when the main requirement is maintaining the result of a transformation or analytical query.
A simple rule I use is:
Streaming Table = how data arrives and moves
Materialized View = how transformed results are maintained
There are exceptions, but this distinction makes architecture decisions much easier.
One Important Streaming Consideration
Streaming semantics work best with append oriented sources.
If existing records in the streaming source are modified or deleted, the pipeline needs an appropriate strategy for handling those changes.
Databricks provides options such as SKIPCHANGECOMMITS for specific scenarios, while CDC workloads can use dedicated change data capture patterns. (Databricks Documentation)
Understanding this behavior is important before simply converting every pipeline into streaming.
Final Thoughts
Materialized Views and Streaming Tables are not competitors.
They are complementary components.
A well designed Databricks architecture can use Streaming Tables to efficiently ingest and incrementally process incoming data while using Materialized Views to maintain cleaned, aggregated, and analytics ready datasets.
The key is to choose the object based on the behavior the workload requires rather than simply choosing between โbatchโ and โstreaming.โ
For Data Engineers, understanding this distinction becomes increasingly important as Databricks continues moving toward declarative pipeline development with Lakeflow and Spark Declarative Pipelines. (Databricks Documentation)
What patterns are you currently using for Materialized Views and Streaming Tables in your Databricks pipelines?
.jpg){kind=link}