Task Hanging issue on DBR 15.4

[Dharma25]

Hello,

I am running strucutred streaming pipeline with 5 models loaded using pyfunc.spark_udf. Lately we have been noticing very strange issue of tasks getting hanged and batch is taking very long time finishing its execution.

CPU utilization is around 90% and Memory utilization is steady.

Issue:

 

Configs:
DBR 15.4
Job Compute
1 driver and 4 workers

[bianca_unifeye]

On DBR 15.4 the DeadlockDetector: TASK_HANGING message usually just means Spark has noticed some very long-running tasks and is checking for deadlocks. With multiple pyfunc.spark_udf models in a streaming query the tasks often appear “stuck” because the Python UDF is blocking (heavy model inference, external calls, or GIL contention) while CPU stays high and memory steady.

I’d suggest:
– checking the Structured Streaming metrics to see if the batch is still progressing,
– taking executor thread dumps to confirm threads are blocked inside the UDF,
– testing the pipeline with fewer models / simplified UDFs to isolate which one causes the hang,
– making sure models are loaded once per executor and not doing network/I/O per row, and, if possible, moving to vectorised / Pandas UDFs.

If the same code works on an older LTS runtime ( try to run on 14.3 or even an older one) but hangs on 15.4, it may be a runtime regression and worth raising with Databricks Support including the job and run IDs.

[Dharma25]

Thank you very much for your recommendations.

Additionally, I noticed that each executor typically has 32 active tasks by default. However, when looking at the test execution summary tab under the DAG for various stages, it displays 300 tasks.

Moreover, I found that executing `coalesce(1)` and then distributing it across all 10 models significantly improves performance, with batches running much faster.