Best practices for optimizing Spark jobs
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā09-30-2024 11:46 PM
What are some best practices for optimizing Spark jobs in Databricks, especially when dealing large datasets? Any tips or resources would be greatly appreciated! Iām trying to analyze data on restaurant menu prices so that insights would be especially helpful!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā10-01-2024 03:12 AM
There are so many.
Here are a few:
- look for data skew
- shuffle as less as possible
- avoid many small files
- use spark and not only pure python
- if using an autoscale cluster: check if you don't lose a lot of time scaling up/down
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā07-15-2025 02:12 AM
Good one @john34567 , made me chuckle but still this is a spam š
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā11-20-2025 03:33 PM
Optimizing Spark jobs is all about using smart data strategies like minimizing shuffles, tuning partitions, caching only what truly matters, and choosing the right file format to keep workloads efficient and cost-effective, and it reminds me of how planning ahead works just like checking the Wetherspoons kids menu before ordering so everything runs smoother, faster, and without unnecessary delays.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ā11-21-2025 12:56 AM
In addition to above cool comments, try to use clusters with VMs enabled for disk caching as well. This caches data at parquet files level in VM local storage, acting as a great complement to spark caching.
