Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Showing results for 
Search instead for 
Did you mean: 

OPTIMIZE command failed to complete on partitioned dataset

New Contributor

Trying to optimize delta table with following stats:

  • size: 212,848 blobs, 31,162,417,246,985 bytes
  • command: OPTIMIZE <table> ZORDER BY (X, Y, Z)

In Spark UI I can see all work divided to batches, and each batch start with 400 tasks to collect data. But each batch processing stage fails after collecting data. Error example:

  • Description: (Batch 11 [ Processing Files ((35651 - 39203) / 213211) ]) Optimizing 3553 files in abfss://<table>
  • Tasks: 1510/3200
  • Failure Reason: Job aborted due to stage failure: Total size of serialized results of 1511 tasks (4.0 GiB) is bigger than spark.driver.maxResultSize 4.0 GiB.

Command runs with default configuration.

Question is why OPTIMIZE process chooses batches that doesn't match limit for spark.driver.maxResultSize? How can we configure splitting to create smaller batches?


Esteemed Contributor III

can you share some sample datasets for this by that we can debug and help you accordingly



Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!