Why does disabling Photon fix my ConcurrentDeleteDeleteException in Databricks?

bharathelsker — Wed, 06 Aug 2025 18:27:54 GMT

I’m running a Databricks 15.4 LTS job with Photon acceleration enabled.

I have a wrapper notebook that uses ThreadPoolExecutor to trigger multiple child notebooks in parallel.
Each thread calls a function that runs a child notebook and updates an audit table (Delta format) with the run status.

This job runs once daily and has worked fine for the past 4 months. However, since last week, the UPDATE statement to the audit table has started throwing a ConcurrentDeleteDeleteException.

I tried several fixes without success, but disabling Photon acceleration seems to solve the problem.

Given that Photon was enabled and working fine last month, can anyone explain:

Why disabling Photon would fix this issue?
Whether there have been any recent changes in Databricks or Delta that could cause this?

Re: Why does disabling Photon fix my ConcurrentDeleteDeleteException in Databricks?

Sidhant07 — Fri, 22 Aug 2025 08:09:13 GMT

Hi @bharathelsker ,

ConcurrentDeleteDeleteException

This exception occurs when a concurrent operation deleted a file that your operation also deletes. This could be caused by two concurrent compaction operations rewriting the same files.

To further investigate why disabling photon fixed the issue, we may need additional details like notebook code and logs and recommend opening a ticket with us.

https://kb.databricks.com/delta-live-tables/dlt-pipeline-failing-with-concurrentdeletedeleteexception-error

topic Why does disabling Photon fix my ConcurrentDeleteDeleteException in Databricks? in Data Engineering

Why does disabling Photon fix my ConcurrentDeleteDeleteException in Databricks?

Re: Why does disabling Photon fix my ConcurrentDeleteDeleteException in Databricks?

Hi @bharathelsker ,

ConcurrentDeleteDeleteException