cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Does anyone know why the optimize does not complete?

irfanaziz
Contributor II

I feel there is some issue with a few partitions of the delta file. The optimize runs fine and completes within few minutes for other partitions but for this particular partition the optimize keeps running forever.

OPTIMIZE delta.`/mnt/prod-abc/Inis/data/comm.delta/` where date='2020-03-05'

Is there a way i could check what is going on or any way to check if there are any issues with the files in this partition or in the deltalog?

I can query the table and i can read this partition but i cannot write this particular partition back to another path(the write keeps running forever). I am running optimize as merge command has been timing out on this delta file. I feel it was a 'z order by(id)' which screwed the distribution. Now i am trying to optimize without the z order.

3 REPLIES 3

Kaniz
Community Manager
Community Manager

Hi @ irfanaziz! My name is Kaniz, and I'm a technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the Forum have an answer to your questions first. Or else I will follow up shortly with a response.

irfanaziz
Contributor II

We had to remove the partition and insert the data again to fix the issue.

Anonymous
Not applicable

@nafri A​ - Thank you for letting us know.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!