Set time out or Auto termination for long running query
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-19-2024 09:37 PM
Hi ,
We want to set auto termination for long running queries in data bricks adhoc cluster.
I attempted below two approaches in my notebook. Despite my understanding that queries should automatically terminate after one hour, with both the approaches queries exceeded this timeout limit and kept running
Approache 1:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-20-2024 02:20 AM
Hi @NehaR
Apply these settings at the cluster-level configuration in the Databricks UI:
Go to the Cluster Settings.
Add the following Spark configuration:
spark.databricks.queryWatchdog.enabled true
spark.databricks.queryWatchdog.timeout 3600
Restart the cluster to apply the settings.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-14-2024 01:24 PM
Hey @JissMathew,
I tried those configs on all-purpose clusters, but to no success. Is there any mention in docs that it is supposed to work?
I know SET STATEMENT_TIMEOUT = 3600 should function only for Warehouses, but I'm looking for a solution that will affect normal clusters.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-16-2024 03:53 AM
Hi @PiotrM For normal all-purpose clusters, you would need an alternative approach to manage query timeouts, as the configuration for STATEMENT_TIMEOUT does not work directly.
Use spark.sql.broadcastTimeout or spark.sql.execution.timeout
spark.conf.set("spark.sql.broadcastTimeout", "3600")
spark.conf.set("spark.sql.execution.timeout", "3600000")
![](/skins/images/97567C72181EBE789E1F0FD869E4C89B/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/97567C72181EBE789E1F0FD869E4C89B/responsive_peak/images/icon_anonymous_message.png)