Does anyone know why the optimize does not complete?

irfanaziz — Wed, 11 Aug 2021 07:59:10 GMT

I feel there is some issue with a few partitions of the delta file. The optimize runs fine and completes within few minutes for other partitions but for this particular partition the optimize keeps running forever.

OPTIMIZE delta.`/mnt/prod-abc/Inis/data/comm.delta/` where date='2020-03-05'

Is there a way i could check what is going on or any way to check if there are any issues with the files in this partition or in the deltalog?

I can query the table and i can read this partition but i cannot write this particular partition back to another path(the write keeps running forever). I am running optimize as merge command has been timing out on this delta file. I feel it was a 'z order by(id)' which screwed the distribution. Now i am trying to optimize without the z order.

Re: Does anyone know why the optimize does not complete?

irfanaziz — Mon, 06 Sep 2021 06:22:25 GMT

We had to remove the partition and insert the data again to fix the issue.

Re: Does anyone know why the optimize does not complete?

Anonymous — Thu, 09 Sep 2021 17:00:50 GMT

@nafri A - Thank you for letting us know.

topic Re: Does anyone know why the optimize does not complete? in Data Engineering

Does anyone know why the optimize does not complete?

Re: Does anyone know why the optimize does not complete?

Re: Does anyone know why the optimize does not complete?