On DBR 15.4 the DeadlockDetector: TASK_HANGING message usually just means Spark has noticed some very long-running tasks and is checking for deadlocks. With multiple pyfunc.spark_udf models in a streaming query the tasks often appear “stuck” because the Python UDF is blocking (heavy model inference, external calls, or GIL contention) while CPU stays high and memory steady.

I’d suggest:
– checking the Structured Streaming metrics to see if the batch is still progressing,
– taking executor thread dumps to confirm threads are blocked inside the UDF,
– testing the pipeline with fewer models / simplified UDFs to isolate which one causes the hang,
– making sure models are loaded once per executor and not doing network/I/O per row, and, if possible, moving to vectorised / Pandas UDFs.

If the same code works on an older LTS runtime ( try to run on 14.3 or even an older one) but hangs on 15.4, it may be a runtime regression and worth raising with Databricks Support including the job and run IDs.

Thank you very much for your recommendations.

Additionally, I noticed that each executor typically has 32 active tasks by default. However, when looking at the test execution summary tab under the DAG for various stages, it displays 300 tasks.

Moreover, I found that executing `coalesce(1)` and then distributing it across all 10 models significantly improves performance, with batches running much faster.