topic SQL Warehouse Timeout / Prevent Long Running Queries in Data Engineering

SQL Warehouse Timeout / Prevent Long Running Queries

Kayla — Tue, 04 Jun 2024 16:46:34 GMT

We have an external service connecting to a SQL Warehouse, running a query that normally lasts 30 minutes.
On occasion an error occurs and it will run for 6 hours.
This happens overnight and is contributing to a larger bill. Is there any way to force long running queries to time out or the SQL Warehouse to have a limited keep-alive time even if there are active queries?

Re: SQL Warehouse Timeout / Prevent Long Running Queries

lucasrocha — Tue, 04 Jun 2024 22:57:18 GMT

Hello @Kayla , I hope this message finds you well.

Have you tried the STATEMENT_TIMEOUT?
The STATEMENT_TIMEOUT configuration parameter sets a timeout value in seconds. Any Databricks SQL statement with a wall clock run time exceeding the value is timed out.
You can set this parameter at the session level using the SET statement and at the global level using Configure SQL parameters or SQL Warehouses API.

doc.: https://docs.databricks.com/en/sql/language-manual/parameters/statement_timeout.html#statement_timeout

Best regards,
Lucas Rocha

Re: SQL Warehouse Timeout / Prevent Long Running Queries

raphaelblg — Tue, 04 Jun 2024 22:57:30 GMT

Hello @Kayla, maybe the STATEMENT_TIMEOUT configuration parameter is what you're looking for.

This parameter can be set at session or global level.

Re: SQL Warehouse Timeout / Prevent Long Running Queries

Kayla — Wed, 05 Jun 2024 11:10:05 GMT

@lucasrocha @raphaelblg That is exactly what I was hoping to find. Thank you!