Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-25-2024 01:18 PM
Hi @MikeGo ,
In general the MERGE statement can be memory-intensive.
To optimize check this article:
https://medium.com/@abhinav.prakash1804/databricks-a-comprehensive-optimization-guide-7b803289c5a7
If you need an additional help, could you write:
1. Are you partitioning/clustering the data by the columns used in JOIN? If yes, how many partitions do you have?
2. What is the average file size of source and target (run DESCRIBE DETAILS on source and target tables)?
3. When doing the merge to the target of 220 million, how many records are in the source?
On Medium, anyone can share insightful perspectives, useful knowledge, and life wisdom with the world.