Thank you very much, @Louis_Frolio , for such a detailed and insightful answer!

All tables used in this processing are managed Delta tables loaded through Unity Catalog.

I will try running it with spark.databricks.io.cache.enabled set to false just to see if the execution plan looks different. I believe we previously tried using REFRESH, but I will attempt to use it again directly after writing changes to the problematic table.