Explanation

Static partition pruning does not work in streaming reads (readStream) by design.

In streaming, Spark:

Can prune files/partitions to some extent

Cannot prune rows inside files

Must scan entire files of selected partitions

Filters like processing_date >= current_date() - n are applied after data is read, not at scan time.

In DLT, current_date() is non-deterministic, which further limits pruning.

What query profile confirms

Partitions read < total partitions → file pruning happened

Rows read still very high → full file scan

Rows returned = 0 → filter applied post-scan

Key takeaway

Partition pruning in Delta streaming works only at file selection level, not row level.

Large historical files inside partitions will still be fully scanned.

Recommended approaches

Use batch read for Bronze → Silver if partition pruning is required

Use Auto Loader to read only new files

Avoid using readStream when selective historical reads are needed