Databricks Community

bampo

Each merge/update on a table with liquid clustering force the streaming to read whole table.Databricks Runtime: 14.3 LTSBelow I prepare a simple scripts to reproduce the issue:Create schema. %sql CREATE SCHEMA IF NOT EXISTS test; Create table with si...

bampo

Thanks for the reply,In the example checkpoint is not used for simplification, but in real scenario it is used for sure.Described scenario reproduces with a single trigger of a stream (without re-triggering the process), so checkpoint is not an issue...

bampo

Thanks for the reply!But it doesn't explain why after the first update whole table is returned (10k records), and after second update (of the same data) only updated records are returned (1k records).Also worth to mention, that in my example I use sm...

Databricks Community

User Stats

User Activity

Streaming Reads Full Table with Liquid Clustering

Re: Streaming Reads Full Table with Liquid Clustering

Re: Streaming Reads Full Table with Liquid Clustering