topic Re: Handling Over-Usage of Capacity in Databricks Jobs/Processes in Get Started Discussions

Handling Over-Usage of Capacity in Databricks Jobs/Processes

smanda88 — Mon, 17 Feb 2025 15:02:39 GMT

Hi all,

Is there a tool or method in Databricks to ensure data integrity and stability when a job or process exceeds the allocated capacity? Specifically, I’m looking for ways to:

Prevent failures or data loss due to resource overuse.
Automatically scale or manage workloads efficiently.
Get alerts or take preventive actions before hitting capacity limits.

Re: Handling Over-Usage of Capacity in Databricks Jobs/Processes

Alberto_Umana — Mon, 17 Feb 2025 15:26:25 GMT

Hello @smanda88 -

For point 1, please see; https://docs.databricks.com/en/lakehouse-architecture/reliability/best-practices.html

For 2, you can use auto-scaling, please refer to: https://docs.databricks.com/en/lakehouse-architecture/cost-optimization/best-practices.html#2-dynamically-allocate-resources

For 3, you can set up job notification.