OPTIMIZE throws an error after doing MERGE on the table

[Constantine]

I have a table on which I do upsert i.e.

MERGE INTO table_name ...

After which I run

OPTIMIZE table_name

Which throws an error

java.util.concurrent.ExecutionException: io.delta.exceptions.ConcurrentDeleteReadException: This transaction attempted to read one or more files that were deleted (for example part-00000-50e8fcea-1314-445b-a4fd-a7b61a9bf02c-c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again.

Not sure, what's happening as I am not deleting any files. Is there a way to fix this?

[Kaniz_Fatma]

Hi @John Constantine​ ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Hubert-Dudek · Accepted solution]

• You can try to change isolation level:

https://docs.microsoft.com/en-us/azure/databricks/delta/optimizations/isolation-level

• In merge is good to specify all partitions in merge conditions.
• It can also happen when script is running concurrently.

[Kaniz_Fatma]

Hi @John Constantine​ , Were you able to troubleshoot the error with the help of @Hubert Dudek​ 's reply?