cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Job Cluster best practices for production workloads

Phani1
Valued Contributor II

 

Hi All,

Can you please share the best practices for job clusters configurations for production workloads

and which is good when compared to serverless and job cluster in production in terms of cost and performance?

Regards,

Phani

 

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @Phani1, For configuring job clusters for production workloads in Databricks, follow these best practices: match cluster size to workload needs, enable autoscaling for dynamic adjustment of worker nodes, use spot instances with a fallback to on-demand for cost savings, leverage cluster pools to minimize startup time, set an idle timeout to shut down unused clusters, monitor performance with tools like Datadog or Azure Monitor, and ensure security with Databricks-backed secret scopes and network configurations. While serverless clusters are cost-effective for sporadic workloads due to their autoscaling capability, job clusters with spot instances and autoscaling are generally more suitable for consistent, high-volume workloads, offering better performance and cost management.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group