Material View to External Delta Table using sink api
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-25-2025 07:17 AM
Hi Team,
While executing the below code i am able to create the sink and my data is getting written into delta tables from materialized view.
import dlt
@Dlt.table(name = "employee_bronze3")
def create_table():
df = spark.read.table("dev.default.employee")
return df
@Dlt.table(name = "employee_silver3")
def create_table():
df = dlt.read("dev.default.employee_bronze3")
return df
dlt.create_sink(
name = "delta_sink",
format = "delta",
options = {"tableName": "dev.default.employee_test"}
)
@Dlt.append_flow(name = "delta_sink_flow", target="delta_sink")
def delta_sink_flow():
return(
dlt.read_stream("dev.default.employee_silver3")
)
But if i already have a materialized view created from different dlt and execute the below statement i am getting the error The sink created is giving error due to the below reason : Cannot stream from Materialized View `dev`.`default`.`employee_silver3`. Streaming from Materialized Views is not supported but why am i not getting the error in the above statement.
dlt.create_sink(
name = "delta_sink",
format = "delta",
options = {"tableName": "dev.default.employee_test"}
)
@Dlt.append_flow(name = "delta_sink_flow", target="delta_sink")
def delta_sink_flow():
return(
spark.readStream.table("dev.default.employee_silver3") , I have also tried with spark.read.table("dev.default.employee_silver3")
)
Can anyone please let me know why this is happening.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2025 03:50 PM
Hi nkrom456,
How are you doing today? as per my understanding, when you use dlt.read_stream() inside the same DLT pipeline, Databricks allows it to stream from that materialized view because everything is being managed within one pipeline — it understands the full flow. But when you try to use spark.readStream.table() to read a materialized view that was created by a different DLT pipeline, it throws an error because streaming from materialized views isn’t supported across pipelines. That’s a Databricks limitation right now. A better way to handle this is to have the first pipeline write to a regular Delta table instead of a materialized view, and then stream from that Delta table in your second pipeline. This keeps things simple and supported. Let me know if you want help updating the code!