Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA teams

Charansai — Fri, 25 Jul 2025 17:38:53 GMT

Hi Community,

We are working on implementing Databricks cluster policies across our organization and are seeking advice on best practices to enforce governance, security, and cost control across different environments.

We have two main teams using Databricks across multiple environments:

Data Engineering – Dev / QA / Prod
Data & Analytics – Dev / QA / Prod

Each environment has a separate Databricks workspace. Our goal is to define robust cluster policies that:

Enforce configuration standards (e.g., disallow public IPs, enforce autoscaling, fixed Spark configs)
Control costs (e.g., limit max workers/memory in dev/QA)
Ensure production stability (e.g., disallow in it scripts or spot instances in prod)
Allow safe experimentation in dev while keeping strong guardrails

Trying to decide:
1. Should we define one policy per team per environment (e.g., data-engineering, analytics) or have general reusable policies for each environment type?
2. What are common policy restrictions used in Dev/QA vs. Prod?
  (e.g., disallowing public IPs, enforcing autoscaling, limiting worker sizes, etc.)
3. Are there any example templates or reusable patterns followed in other large organizations?
4. Any tips for balancing developer flexibility with platform governance?
5. Please differentiate between data engineers and data analytics across all environments and provide the code for it.
We appreciate any advice, templates, or governance experiences you can share!
Thanks in advance!

Re: Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA tea

ilir_nuredini — Wed, 30 Jul 2025 13:24:36 GMT

Hello @Charansai ,

You may check this resource by Databricks:
Databricks Workspace Administration Guide | Databricks Blog
Best practice articles - Azure Databricks | Microsoft Learn

Even though doesn't answer all of your points, I hope it will help you on deciding for some of them.

Best, Ilir

Re: Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA tea

Charansai — Wed, 30 Jul 2025 14:48:21 GMT

I just want to confirm one more thing here is that me as admin managing the cluster creation and no user will have access to create them me know how the cluster policies help me in this perspective.

topic Re: Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA tea in Community Articles

Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA teams

Re: Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA tea

Re: Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA tea