WiliamRosa
Databricks Partner

If batch jobs on Databricks suddenly slow down without a change in data volume, the root cause could be shifting resource contention, skewed data partitions, or Spark-level bottlenecks. Checking the Spark UI’s Jobs and Stages views for imbalanced tasks, long-running stages, or high shuffle and garbage collection times can reveal hotspots. It may also help to right-size the cluster via dynamic allocation, address data skew or small-file issues, and review recent cluster usage. Enabling features like predictive optimization, compaction, and caching, or even shifting to serverless compute to bypass cold-start delays, can also restore performance consistency.

Wiliam Rosa
Data Engineer | Machine Learning Engineer
LinkedIn: linkedin.com/in/wiliamrosa