OPTIMIZE throws an error after doing MERGE on the table

Constantine
Contributor III

I have a table on which I do upsert i.e.

MERGE INTO table_name ...

After which I run

OPTIMIZE table_name

Which throws an error

java.util.concurrent.ExecutionException: io.delta.exceptions.ConcurrentDeleteReadException: This transaction attempted to read one or more files that were deleted (for example part-00000-50e8fcea-1314-445b-a4fd-a7b61a9bf02c-c000.snappy.parquet in the root of the table) by a concurrent update. Please try the operation again.

Not sure, what's happening as I am not deleting any files. Is there a way to fix this?

Hubert-Dudek
Databricks MVP
  • You can try to change isolation level:

https://docs.microsoft.com/en-us/azure/databricks/delta/optimizations/isolation-level

  • In merge is good to specify all partitions in merge conditions.
  • It can also happen when script is running concurrently.

My blog: https://databrickster.medium.com/

View solution in original post