Materialized view always load full table instead of incremental
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2025 11:42 PM
My delta table are stored at HANA data lake file and I have ETL configured like below
@DP.materialized_view(temporary=True)
def source():
return spark.read.format("delta").load("/data/source")
@dp.materialized_view
def sink():
return spark.read.table("source").withColumnRenamed("COL_A", "COL_B")When I first ran pipeline, it show 100k records has been processed for both table.
For the second run, since there is no update from source table, so I'm expecting no records will be processed. But the dashboard still show 100k.
I'm also check whether the source table enable change data feed by executing
dt = DeltaTable.forPath(spark, "/data/source")
detail = dt.detail().collect()[0]
props = detail.asDict().get("properties", {})
for k, v in props.items():
print(f"{k}: {v}")and the result is
pipelines.metastore.tableName: `default`.`source`
pipelines.pipelineId: 645fa38f-f6bf-45ab-a696-bd923457dc85
delta.enableChangeDataFeed: trueAnybody knows what am I missing here?
Thank in advance.