I’m running a Databricks 15.4 LTS job with Photon acceleration enabled.

I have a wrapper notebook that uses ThreadPoolExecutor to trigger multiple child notebooks in parallel.
Each thread calls a function that runs a child notebook and updates an audit table (Delta format) with the run status.

This job runs once daily and has worked fine for the past 4 months. However, since last week, the UPDATE statement to the audit table has started throwing a ConcurrentDeleteDeleteException.

I tried several fixes without success, but disabling Photon acceleration seems to solve the problem.

Given that Photon was enabled and working fine last month, can anyone explain:

1. Why disabling Photon would fix this issue?

2. Whether there have been any recent changes in Databricks or Delta that could cause this?