Job Cluster best practices for production workloads

Phani1 — Tue, 30 Jul 2024 10:47:34 GMT

Hi All,

Can you please share the best practices for job clusters configurations for production workloads

and which is good when compared to serverless and job cluster in production in terms of cost and performance?

Regards,

Phani

Re: Job Cluster best practices for production workloads

Retired_mod — Wed, 07 Aug 2024 13:41:07 GMT

Hi @Phani1, For configuring job clusters for production workloads in Databricks, follow these best practices: match cluster size to workload needs, enable autoscaling for dynamic adjustment of worker nodes, use spot instances with a fallback to on-demand for cost savings, leverage cluster pools to minimize startup time, set an idle timeout to shut down unused clusters, monitor performance with tools like Datadog or Azure Monitor, and ensure security with Databricks-backed secret scopes and network configurations. While serverless clusters are cost-effective for sporadic workloads due to their autoscaling capability, job clusters with spot instances and autoscaling are generally more suitable for consistent, high-volume workloads, offering better performance and cost management.

topic Job Cluster best practices for production workloads in Get Started Discussions

Job Cluster best practices for production workloads

Re: Job Cluster best practices for production workloads