Why does disabling Photon fix my ConcurrentDeleteDeleteException in Databricks?

[bharathelsker]

I’m running a Databricks 15.4 LTS job with Photon acceleration enabled.

I have a wrapper notebook that uses ThreadPoolExecutor to trigger multiple child notebooks in parallel.
Each thread calls a function that runs a child notebook and updates an audit table (Delta format) with the run status.

This job runs once daily and has worked fine for the past 4 months. However, since last week, the UPDATE statement to the audit table has started throwing a ConcurrentDeleteDeleteException.

I tried several fixes without success, but disabling Photon acceleration seems to solve the problem.

Given that Photon was enabled and working fine last month, can anyone explain:

1. Why disabling Photon would fix this issue?

2. Whether there have been any recent changes in Databricks or Delta that could cause this?

[Sidhant07 · Accepted solution]

Hi @bharathelsker ,

ConcurrentDeleteDeleteException

This exception occurs when a concurrent operation deleted a file that your operation also deletes. This could be caused by two concurrent compaction operations rewriting the same files.

To further investigate why disabling photon fixed the issue, we may need additional details like notebook code and logs and recommend opening a ticket with us.

https://kb.databricks.com/delta-live-tables/dlt-pipeline-failing-with-concurrentdeletedeleteexceptio...