balajij8
Contributor III

You can handle it as below

  • Fix the Bronze Write - The 20+ minutes commit gap suggests metadata contention or "Small File Issues" in the bronze delta tables. You can optimize tables manually or enable Optimized Write and Auto Optimize if feasible. This ensures the Spark driver balances data across executors to write larger more efficient Parquet files significantly reducing the number of entries the Delta Log needs to commit.
  • Streaming Jobs to avoid Look Back: You can use streaming jobs to avoid maintaining Look Back & audit table as it automatically uses the checkpoint to find the unprocessed records from bronze