TimeoutException: Stream Execution thread for stream [xxxxxx]failed to stop within 15000 millisecond
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-26-2024 10:05 AM
TimeoutException: Stream Execution thread for stream [id = xxx runId = xxxx] failed to stop within 15000 milliseconds (specified by spark.sql.streaming.stopTimeout). See the cause on what was being executed in the streaming query thread.
I have a databricks notebook which is triggered via ADF pipeline, it loads data from landing to azure delta and export to teradata tables.
Since last thurdsday am seeing the timeout expection
Databricks Runtime : 9.1 LTS Photon (includes Apache Spark 3.1.2, Scala 2.12)
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-27-2024 09:05 AM
@User_1611 - could you please try the following ?
- Reduce the number of streaming queries running on the same cluster
- Make sure your code does not try to re-trigger/start an active streaming query
- Make sure to collect the thread dumps if this error happens again in the future.
