- 8402 Views
- 5 replies
- 0 kudos
Optimize & Compaction
Hi,From which data bricks runtime will support Optimize and compaction
- 8402 Views
- 5 replies
- 0 kudos
- 0 kudos
Optimize and compaction are operations commonly used in Apache Spark for optimizing and improving the performance of data storage and processing. Databricks, which is a cloud-based platform for Apache Spark, provides support for these operations on v...
- 0 kudos
- 2651 Views
- 3 replies
- 1 kudos
How can I optimize my data pipeline?
Delta Lake provides optimizations that can help you accelerate your data lake operations. Here’s how you can improve query speed by optimizing the layout of data in storage.There are two ways you can optimize your data pipeline: 1) Notebook Optimizat...
- 2651 Views
- 3 replies
- 1 kudos
- 1 kudos
some tips from me:Look for data skews; some partitions can be huge, some small because of incorrect partitioning. You can use Spark UI to do that but also debug your code a bit (get getNumPartitions()), especially SQL can divide it unequally to parti...
- 1 kudos