Terraform folder structure and states

[ismaelhenzel]

Hi everyone, I have a few questions regarding Terraform folder structure and state management for Databricks, and I'd love to get your opinions.

For context, our Databricks environment is already deployed and Unity Catalog is configured. The Terraform code I’m working on is strictly for managing workspace-level resources.

Question 1: After creating and calling modules for clusters, SQL warehouses, and cluster policies, my main.tf file has become very large. Should I split these into separate, flattened files by resource type (e.g., clusters.tf, secrets.tf, policies.tf)?

Question 2: Currently, I am also managing Unity Catalog resources (catalogs, schemas, volumes, external locations, and their respective grants) within this same workspace-level state. As the environment grows, the state file is becoming quite large. I am considering separating the Unity Catalog resources into their own isolated Terraform state. Does this approach make sense?"

[Ashwin_DSA · Accepted solution]

Hi @ismaelhenzel,

In terms of your first question, Terraform automatically loads all *.tf files in the same directory, so it’s common practice to organise them by concern. For example,

envs/
  prod/
    backend.tf          # remote state config
    providers.tf        # databricks + cloud providers
    versions.tf
    locals.tf
    clusters.tf         # cluster + pools modules
    sql_warehouses.tf
    cluster_policies.tf
    secrets.tf

This matches Databricks’ general IaC guidance to use modules and clear patterns for "workspace-level configuration" like cluster policies, pools, secrets, etc., while keeping state in a remote backend and using separate states per environment.

For the other question, it’s reasonable (and often desirable) to move Unity Catalog resources (metastore, catalogs, schemas, external locations, storage credentials, grants) into their own state once things grow. UC is a governance/metastore concern, often owned by a different team and with a different change cadence than compute. In the same link I shared above for your first question, you will see that the guidance explicitly calls out Unity Catalog resources as their own deployment pattern alongside account‑level infrastructure and workspace‑level configuration. You can also see that the official UC Terraform guides article shows UC as a coherent unit (metastore, external locations, catalogues, schemas, grants) managed via the Databricks provider, which maps nicely to its own state file/project.

If you want some concrete end‑to‑end UC + Terraform examples to crib from, the official docs and registry guides are good starting points.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[ismaelhenzel]

Thanks for your answer, Ashwin! This really clarifies how to better structure the workspace Terraform. I completely agree with your points, and your approach aligns perfectly with what I had in mind. I will split my main.tf into separate files based on the resources being created in the workspace. I'll also separate out the Unity Catalog resources into their own state. As you mentioned, their change cadence is very different, and they grow the state file much more quickly—especially when adopting strategies like domain-oriented catalogs. Thanks again!