cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

OPTIMIZE throws an error after doing MERGE on the table

Constantine
Contributor III

I have a table on which I do upsert i.e.

MERGE INTO table_name ...

After which I run

OPTIMIZE table_name

Which throws an error

java.util.concurrent.ExecutionException: io.delta.exceptions.ConcurrentDeleteReadException: This transaction attempted to read one or more files that were deleted (for example part-00000-50e8fcea-1314-445b-a4fd-a7b61a9bf02c-c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again.

Not sure, what's happening as I am not deleting any files. Is there a way to fix this?

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III
  • You can try to change isolation level:

https://docs.microsoft.com/en-us/azure/databricks/delta/optimizations/isolation-level

  • In merge is good to specify all partitions in merge conditions.
  • It can also happen when script is running concurrently.

View solution in original post

3 REPLIES 3

Kaniz
Community Manager
Community Manager

Hi @John Constantine​ ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

Hubert-Dudek
Esteemed Contributor III
  • You can try to change isolation level:

https://docs.microsoft.com/en-us/azure/databricks/delta/optimizations/isolation-level

  • In merge is good to specify all partitions in merge conditions.
  • It can also happen when script is running concurrently.

Hi @John Constantine​ , Were you able to troubleshoot the error with the help of @Hubert Dudek​ 's reply?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!