how to manage a dynamic scheduled job if an INTERNAL_ERROR occurs?

[jeremy98]

Hi community,

My team and I have been occasionally experiencing INTERNAL_ERROR events in Databricks. We have a job that runs on a schedule, but the start times vary. Sometimes, when the job is triggered, the underlying cluster fails to start for some reason.

I’d like some advice on how to better investigate these issues and how to set up a mitigation or fallback mechanism. Specifically, I want a way to detect when the job starts but the cluster cannot initialize, and then run an alternative process or alert.

Any suggestions or best practices would be greatly appreciated!

[SP_6721]

Hi @jeremy98 ,

To investigate, check the Jobs UI for failed runs and review both error messages and cluster logs. Monitor failure trends over time and adjust cluster settings or quotas if needed.
https://docs.databricks.com/gcp/en/jobs/repair-job-failures

For detection, enable job notifications for “on failure” events in Job settings.
https://docs.databricks.com/gcp/en/jobs/notifications

For fallback, add a downstream task in the job configured to run only if failed.