cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

OPTIMIZE command failed to complete on partitioned dataset

MaximS
New Contributor

Trying to optimize delta table with following stats:

  • size: 212,848 blobs, 31,162,417,246,985 bytes
  • command: OPTIMIZE <table> ZORDER BY (X, Y, Z)

In Spark UI I can see all work divided to batches, and each batch start with 400 tasks to collect data. But each batch processing stage fails after collecting data. Error example:

  • Description: (Batch 11 [ Processing Files ((35651 - 39203) / 213211) ]) Optimizing 3553 files in abfss://<table>
  • Tasks: 1510/3200
  • Failure Reason: Job aborted due to stage failure: Total size of serialized results of 1511 tasks (4.0 GiB) is bigger than spark.driver.maxResultSize 4.0 GiB.

Command runs with default configuration.

Question is why OPTIMIZE process chooses batches that doesn't match limit for spark.driver.maxResultSize? How can we configure splitting to create smaller batches?

1 REPLY 1

Aviral-Bhardwaj
Esteemed Contributor III

can you share some sample datasets for this by that we can debug and help you accordingly

Thanks

Aviral

AviralBhardwaj

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group