cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Notebook runs not found due to retention limits with dbutils.notebook.run

ErikApption
New Contributor II

We saw this odd error in an AWS deployment, we have one notebook calling another one through

 dbutils.notebook.run(...) 

and this suddenly stopped working and failed with "Notebook runs not found due to retention limits", the "learn more" points to Databricks Documentation

Any clues what can be causing this and how to fix it?

 

1 REPLY 1

Isi
Contributor

Hey @ErikApption ,

Maybe I am wrong but I will give you my opinion.

Each time you execute dbutils.notebook.run(), it launches a new and independent execution within the same cluster. So, if you run the cell today and then run it again tomorrow, there should be two separate ephemeral executions.

However, the issue is that Databricks does not treat these as persistent jobs—they are temporary executions that are not stored permanently.

Databricks automatically deletes old executions

•Even though new executions are created each time you run the notebook, Databricks automatically deletes old executions based on its retention policies.
•If a process tries to access a past execution that has already been deleted, the error “Notebook runs not found due to retention limits” will appear.

Timeout or Expiry Constraints

dbutils.notebook.run() allows a timeout_seconds parameter, but if Databricks is down for more than 10 minutes, the run fails regardless of timeout settings
If no timeout is set (timeout_seconds=0), the notebook must still complete within 30 days, otherwise, it gets removed.

Job Execution History Not Tracked
• Unlike standard Databricks Jobs, which persist in the Jobs UI, notebook runs triggered via dbutils.notebook.run() do not retain long-term execution logs.
•Once the metadata is deleted, Databricks can no longer retrieve details about past executions, leading to this error.

Hope have given you an idea 🙂

Isi

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group