OPTIMIZE command failed to complete on partitioned dataset

MaximS — Fri, 16 Dec 2022 15:25:35 GMT

Trying to optimize delta table with following stats:

size: 212,848 blobs, 31,162,417,246,985 bytes
command: OPTIMIZE <table> ZORDER BY (X, Y, Z)

In Spark UI I can see all work divided to batches, and each batch start with 400 tasks to collect data. But each batch processing stage fails after collecting data. Error example:

Description: (Batch 11 [ Processing Files ((35651 - 39203) / 213211) ]) Optimizing 3553 files in abfss://<table>
Tasks: 1510/3200
Failure Reason: Job aborted due to stage failure: Total size of serialized results of 1511 tasks (4.0 GiB) is bigger than spark.driver.maxResultSize 4.0 GiB.

Command runs with default configuration.

Question is why OPTIMIZE process chooses batches that doesn't match limit for spark.driver.maxResultSize? How can we configure splitting to create smaller batches?

Re: OPTIMIZE command failed to complete on partitioned dataset

Aviral-Bhardwaj — Sun, 18 Dec 2022 06:49:34 GMT

can you share some sample datasets for this by that we can debug and help you accordingly

Thanks

Aviral

topic OPTIMIZE command failed to complete on partitioned dataset in Data Engineering

OPTIMIZE command failed to complete on partitioned dataset

Re: OPTIMIZE command failed to complete on partitioned dataset