Best Practices for Optimizing Delta Lake Performance
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-18-2024 01:18 AM
I’m currently working on a project using Delta Lake on Databricks and would like to optimize performance for both batch and streaming workloads.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-18-2024 04:21 AM - edited 09-18-2024 04:25 AM
Hi @joshbuttler ,
It's a big topic and it really depends on the use case. You can follow some guides/recommendations from databricks documentation:
- Optimization recommendations on Databricks | Databricks on AWS
- Use liquid clustering for Delta tables | Databricks on AWS
- Comprehensive Guide to Optimize Data Workloads | Databricks
And if your company is databricks partner then you have access to a nice course on optimization in Databricks Academy for free.
