Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA teams

[Charansai]

Hi Community,

We are working on implementing Databricks cluster policies across our organization and are seeking advice on best practices to enforce governance, security, and cost control across different environments.

We have two main teams using Databricks across multiple environments:

1. Data Engineering – Dev / QA / Prod

2. Data & Analytics – Dev / QA / Prod

Each environment has a separate Databricks workspace. Our goal is to define robust cluster policies that:

•  Enforce configuration standards (e.g., disallow public IPs, enforce autoscaling, fixed Spark configs)

• Control costs (e.g., limit max workers/memory in dev/QA)

• Ensure production stability (e.g., disallow in it scripts or spot instances in prod)

• Allow safe experimentation in dev while keeping strong guardrails
  
  Trying to decide:

  1. Should we define one policy per team per environment (e.g., data-engineering, analytics) or have general reusable policies for each environment type?

  2. What are common policy restrictions used in Dev/QA vs. Prod?
     (e.g., disallowing public IPs, enforcing autoscaling, limiting worker sizes, etc.)

  3. Are there any example templates or reusable patterns followed in other large organizations?

  4. Any tips for balancing developer flexibility with platform governance?
     
     

  5. Please differentiate between data engineers and data analytics across all environments and provide the code for it.

  We appreciate any advice, templates, or governance experiences you can share!

  Thanks in advance!

[ilir_nuredini]

Hello @Charansai ,

You may check this resource by Databricks: 
Databricks Workspace Administration Guide | Databricks Blog
Best practice articles - Azure Databricks | Microsoft Learn

Even though doesn't answer all of your points, I hope it will help you on deciding for some of them.

Best, Ilir

[Charansai]

I just want to confirm one more thing here is that me as admin managing the cluster creation and no user will have access to create them me know how the cluster policies help me in this perspective.