topic Re: Set time out or Auto termination for long running query in Data Engineering

Set time out or Auto termination for long running query

NehaR — Wed, 20 Nov 2024 05:37:29 GMT

Hi ,

We want to set auto termination for long running queries in data bricks adhoc cluster.

I attempted below two approaches in my notebook. Despite my understanding that queries should automatically terminate after one hour, with both the approaches queries exceeded this timeout limit and kept running

Approache 1:

SET spark.databricks.queryWatchdog.enabled=true;

SET spark.databricks.queryWatchdog.timeout=3600;

Approache 2:

SET STATEMENT_TIMEOUT = 3600;

Please suggest what wrong am i doing here?

Re: Set time out or Auto termination for long running query

JissMathew — Wed, 20 Nov 2024 10:20:27 GMT

Hi @NehaR

Apply these settings at the cluster-level configuration in the Databricks UI:

Go to the Cluster Settings.
Add the following Spark configuration:
spark.databricks.queryWatchdog.enabled true

spark.databricks.queryWatchdog.timeout 3600

Restart the cluster to apply the settings.

Re: Set time out or Auto termination for long running query

PiotrM — Sat, 14 Dec 2024 21:24:47 GMT

Hey @JissMathew,
I tried those configs on all-purpose clusters, but to no success. Is there any mention in docs that it is supposed to work?
I know SET STATEMENT_TIMEOUT = 3600 should function only for Warehouses, but I'm looking for a solution that will affect normal clusters.

Re: Set time out or Auto termination for long running query

JissMathew — Mon, 16 Dec 2024 11:53:19 GMT

Hi @PiotrM For normal all-purpose clusters, you would need an alternative approach to manage query timeouts, as the configuration for STATEMENT_TIMEOUT does not work directly.

Use spark.sql.broadcastTimeout or spark.sql.execution.timeout

spark.conf.set("spark.sql.broadcastTimeout", "3600")
spark.conf.set("spark.sql.execution.timeout", "3600000")

Re: Set time out or Auto termination for long running query

NehaR — Fri, 07 Mar 2025 21:33:02 GMT

Hi @JissMathew ,

Thanks for sharing these settings. I see settings work for sql warehouse clusters and normal all-purpose clusters.

Do we have similar setting option for serverless compute also?

Re: Set time out or Auto termination for long running query

vinaypvsn — Wed, 08 Oct 2025 04:42:18 GMT

Hi @NehaR are the configurations(spark.sql.broadcastTimeout or spark.sql.execution.timeout) working when we set at cluster level. I am currently trying to do a similar configuration for compute clusters but it dosent work.