Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-05-2026 03:03 AM
Hi becasue you need a literal not dynamic expression, with dynanic expression all partitions are already scanned. so btr to use literal. try this:
def filter_latest_records(df, timing_column="processing_date"):
yesterday = spark.sql("select date_sub(current_date(), 1) as d").first()["d"]
return df.filter(col(timing_column) >= yesterday)
