ManojkMohan
Honored Contributor II

@pooja_bhumandla 

  1. Join strategy

Memory and cluster configuration

  • Increase executor memory and memory overhead if the job is genuinely heavy
  • Reduce the number of cores per executor to give each task more memory headroom

3 Micro‑batch load

If the size of the source side of the MERGE grows over time, consider limiting micro‑batch size

https://docs.databricks.com/aws/en/structured-streaming/foreach

If you share your cluster specs (executor memory/cores, threshold settings, and rough sizes of the tables on each side of the MERGE), some more solutioning brainstorming can be done