topic Re: Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster in Data Engineering

Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster

zenwanderer — Sat, 28 Mar 2026 16:41:56 GMT

Hi everyone, I’m facing an issue when running a notebook on a Databricks All-purpose cluster. Some of my cells/pipelines run for a very long time, and I want to automatically cancel/kill them when they exceed a certain time limit.

I tried setting spark.databricks.execution.timeout, but it doesn’t seem to have any effect in my case.

What I need is a timeout mechanism that can cancel the currently running notebook cell, not just a Spark job timeout.

If anyone can share guidance or official documentation references, I’d really appreciate it. Thanks in advance!

Re: Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster

balajij8 — Sat, 28 Mar 2026 18:25:17 GMT

You can use signal to do this if running in a notebook for code validation

#Add in notebook import signal class TimeoutException(Exception): """Raised when a cell is run for very long time""" def timeout_handler(signum, frame): raise TimeoutException("Timed out!") def set_cell_timeout(seconds): signal.signal(signal.SIGALRM, timeout_handler) signal.alarm(seconds) #Add in a notebook cell running notebook function try: set_cell_timeout(30) # Set for 30 seconds #notebook function finally: signal.alarm(0)

You can use lakeflow job notifications with threshold to cancel jobs running too long. Avoid using signals in notebooks running via jobs

Re: Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster

zenwanderer — Sun, 29 Mar 2026 02:20:52 GMT

What I’m looking for is a workspace-level monitoring approach: detect any notebook execution where a cell (or the run) has been running longer than a threshold, and then cancel/terminate it automatically.

I’ve tried looking into audit tables, REST APIs, but it seems they don’t provide enough visibility at cell-level

Re: Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster

balajij8 — Sun, 29 Mar 2026 04:58:09 GMT

For the issue - Some of my cells/pipelines run for a very long time, and I want to automatically cancel/kill them when they exceed a certain time limit.

You can use job notifications with Metric threshold (Duration Warning for notifications & Duration Timeout for kill) to cancel jobs running too long (completion time more than Duration Timeout). More details here

Re: Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster

MoJaMa — Mon, 30 Mar 2026 02:09:19 GMT

@zenwanderer Have you looked into Query Watchdog?

For Classic All-Purpose clusters this might be your best bet.

https://docs.databricks.com/aws/en/compute/troubleshooting/query-watchdog