Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2025 06:25 AM
If batch jobs on Databricks suddenly slow down without a change in data volume, the root cause could be shifting resource contention, skewed data partitions, or Spark-level bottlenecks. Checking the Spark UI’s Jobs and Stages views for imbalanced tasks, long-running stages, or high shuffle and garbage collection times can reveal hotspots. It may also help to right-size the cluster via dynamic allocation, address data skew or small-file issues, and review recent cluster usage. Enabling features like predictive optimization, compaction, and caching, or even shifting to serverless compute to bypass cold-start delays, can also restore performance consistency.
Wiliam Rosa
Data Engineer | Machine Learning Engineer
LinkedIn: linkedin.com/in/wiliamrosa
Data Engineer | Machine Learning Engineer
LinkedIn: linkedin.com/in/wiliamrosa