- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-31-2023 04:10 AM
there are several methods:
you can disable optimizations (see the databricks delta lake performance optimization help files) but I would advise against that.
Databricks default settings of the most recent runtimes are pretty optimized IMO. You can write fast using 80 cpus (so 80 partitions) but that will have a negative performance impact when reading this data.
Semantic Partitioning of the delta table is certainly a good idea (if not already done). And there is also Z-ORDER.
There is no simple answer to this.
If your merge in the end will work in parallel, you also have to take the data skew into account.
Debugging is really hard if almost impossible in spark due to the parallel nature of the application.