Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-22-2026 03:08 AM
Hi,
I was going through the documentation on quarantining records. Initially I thought that partitioning is not supported for temporary tables however I came cross the following
@DP.table(
temporary=True,
partition_cols=["is_quarantined"],
)
@dp.expect_all(rules)
def trips_data_quarantine():
return (
spark.readStream.table("raw_trips_data").withColumn("is_quarantined", expr(quarantine_rules))
)I'm unable to understand how partitioning is been applied for a temporary table. Could anyone please explain this🙂