OPTIMIZE command failed to complete on partitioned dataset

[MaximS]

Trying to optimize delta table with following stats:

• size: 212,848 blobs, 31,162,417,246,985 bytes
• command: OPTIMIZE <table> ZORDER BY (X, Y, Z)

In Spark UI I can see all work divided to batches, and each batch start with 400 tasks to collect data. But each batch processing stage fails after collecting data. Error example:

• Description: (Batch 11 [ Processing Files ((35651 - 39203) / 213211) ]) Optimizing 3553 files in abfss://<table>
• Tasks: 1510/3200
• Failure Reason: Job aborted due to stage failure: Total size of serialized results of 1511 tasks (4.0 GiB) is bigger than spark.driver.maxResultSize 4.0 GiB.

Command runs with default configuration.

Question is why OPTIMIZE process chooses batches that doesn't match limit for spark.driver.maxResultSize? How can we configure splitting to create smaller batches?

[Aviral-Bhardwaj]

can you share some sample datasets for this by that we can debug and help you accordingly

Thanks

Aviral