Task Hanging issue on DBR 15.4

- - Certifications
- - Learning Paths
- - Databricks Product Tours
- - Get Started Guides

- - Get Started Resources
- - Events
- - Support FAQs
- - Technical Blog
- - Community Articles
- - Announcements
- - DatabricksTV
- - Product Platform Updates

- - Private Groups
  - Princeton Life Sciences Databricks User Group
- - Skills@Scale

- - Databricks Community Champions
- - Khoros Community Forums Support (Not for Databricks Product Questions)
- - Databricks Community Code of Conduct

Administration & Architecture

Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.

Hello,

I am running strucutred streaming pipeline with 5 models loaded using pyfunc.spark_udf. Lately we have been noticing very strange issue of tasks getting hanged and batch is taking very long time finishing its execution.

CPU utilization is around 90% and Memory utilization is steady.

Issue:

Configs:
DBR 15.4
Job Compute
1 driver and 4 workers