Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2025 12:56 PM
- Join strategy
For the Delta MERGE, try to ensure the large side is not broadcast by setting autoBroadcastJoinThreshold low or disabling it https://kb.databricks.com/sql/bchashjoin-exceeds-bcjointhreshold-oom
- If you are explicitly broadcasting a reference DataFrame, remove the hint or replace with a shuffle join hint
2 Memory and cluster configuration
- Increase executor memory and memory overhead if the job is genuinely heavy
- Reduce the number of cores per executor to give each task more memory headroom
3 Micro‑batch load
If the size of the source side of the MERGE grows over time, consider limiting micro‑batch size
https://docs.databricks.com/aws/en/structured-streaming/foreach
If you share your cluster specs (executor memory/cores, threshold settings, and rough sizes of the tables on each side of the MERGE), some more solutioning brainstorming can be done