I have created bronze table and partitioned on processing date which is date column. In silver table i am putting filter on basis of processing date column to read last 2 days data but it is reading 37 million data but i have only 24722 in last 2 days partitions. Below is screenshot of query profile in Delta live table
Below is function to filter last 2 days data
def filter_latest_records(df,timing_column = 'processing_date'😞
df = df.filter(col(timing_column) >= date_sub(current_date(), 1))
return df
Can some help me why it is reading 37 million data?
