OPTIMIZE throws an error after doing MERGE on the table

[Constantine]

I have a table on which I do upsert i.e.

MERGE INTO table_name ...

After which I run

OPTIMIZE table_name

Which throws an error

java.util.concurrent.ExecutionException: io.delta.exceptions.ConcurrentDeleteReadException: This transaction attempted to read one or more files that were deleted (for example part-00000-50e8fcea-1314-445b-a4fd-a7b61a9bf02c-c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again.

Not sure, what's happening as I am not deleting any files. Is there a way to fix this?

[Hubert-Dudek · Accepted solution]

• You can try to change isolation level:

https://docs.microsoft.com/en-us/azure/databricks/delta/optimizations/isolation-level

• In merge is good to specify all partitions in merge conditions.
• It can also happen when script is running concurrently.