Notebook runs not found due to retention limits with dbutils.notebook.run

ErikApption — Fri, 14 Mar 2025 18:34:53 GMT

We saw this odd error in an AWS deployment, we have one notebook calling another one through

 dbutils.notebook.run(...)

and this suddenly stopped working and failed with "Notebook runs not found due to retention limits", the "learn more" points to Databricks Documentation

Any clues what can be causing this and how to fix it?

Re: Notebook runs not found due to retention limits with dbutils.notebook.run

Isi — Mon, 17 Mar 2025 21:27:10 GMT

Hey @ErikApption ,

Maybe I am wrong but I will give you my opinion.

Each time you execute dbutils.notebook.run(), it launches a new and independent execution within the same cluster. So, if you run the cell today and then run it again tomorrow, there should be two separate ephemeral executions.

However, the issue is that Databricks does not treat these as persistent jobs—they are temporary executions that are not stored permanently.

Databricks automatically deletes old executions

•Even though new executions are created each time you run the notebook, Databricks automatically deletes old executions based on its retention policies.
•If a process tries to access a past execution that has already been deleted, the error “Notebook runs not found due to retention limits” will appear.

Timeout or Expiry Constraints

•dbutils.notebook.run() allows a timeout_seconds parameter, but if Databricks is down for more than 10 minutes, the run fails regardless of timeout settings
•If no timeout is set (timeout_seconds=0), the notebook must still complete within 30 days, otherwise, it gets removed.

Job Execution History Not Tracked
• Unlike standard Databricks Jobs, which persist in the Jobs UI, notebook runs triggered via dbutils.notebook.run() do not retain long-term execution logs.
•Once the metadata is deleted, Databricks can no longer retrieve details about past executions, leading to this error.

Hope have given you an idea 🙂

Isi

topic Re: Notebook runs not found due to retention limits with dbutils.notebook.run in Administration & Architecture

Notebook runs not found due to retention limits with dbutils.notebook.run

Re: Notebook runs not found due to retention limits with dbutils.notebook.run