Handling Over-Usage of Capacity in Databricks Jobs/Processes

smanda88
New Contributor

‎02-17-2025 07:02 AM

Hi all,

Is there a tool or method in Databricks to ensure data integrity and stability when a job or process exceeds the allocated capacity? Specifically, I’m looking for ways to:

  1. Prevent failures or data loss due to resource overuse.
  2. Automatically scale or manage workloads efficiently.
  3. Get alerts or take preventive actions before hitting capacity limits.
0 Kudos

Alberto_Umana
Databricks Employee
Databricks Employee

‎02-17-2025 07:26 AM

Hello @smanda88 -

For point 1, please see; https://docs.databricks.com/en/lakehouse-architecture/reliability/best-practices.html

For 2, you can use auto-scaling, please refer to: https://docs.databricks.com/en/lakehouse-architecture/cost-optimization/best-practices.html#2-dynami...

For 3, you can set up job notification.

Alberto_Umana_0-1739805977569.png

 

0 Kudos