Hey @ErikApption ,
Maybe I am wrong but I will give you my opinion.
Each time you execute dbutils.notebook.run(), it launches a new and independent execution within the same cluster. So, if you run the cell today and then run it again tomorrow, there should be two separate ephemeral executions.
However, the issue is that Databricks does not treat these as persistent jobs—they are temporary executions that are not stored permanently.
Databricks automatically deletes old executions
•Even though new executions are created each time you run the notebook, Databricks automatically deletes old executions based on its retention policies.
•If a process tries to access a past execution that has already been deleted, the error “Notebook runs not found due to retention limits” will appear.
Timeout or Expiry Constraints
•dbutils.notebook.run() allows a timeout_seconds parameter, but if Databricks is down for more than 10 minutes, the run fails regardless of timeout settings
•If no timeout is set (timeout_seconds=0), the notebook must still complete within 30 days, otherwise, it gets removed.
Job Execution History Not Tracked
• Unlike standard Databricks Jobs, which persist in the Jobs UI, notebook runs triggered via dbutils.notebook.run() do not retain long-term execution logs.
•Once the metadata is deleted, Databricks can no longer retrieve details about past executions, leading to this error.
Hope have given you an idea 🙂
Isi
