topic Re: Managing Spark Declarative Pipelines Permissions in Data Governance

Managing Spark Declarative Pipelines Permissions

nulltype — Mon, 15 Dec 2025 19:29:01 GMT

We are currently managing our permissions via Terraform (including cluster creation, UC governance, etc.). We have a specific `data_engineer` role, and we need everyone with this role to be able to view and manage all of our SDPs.

The Issue: Currently, only Workspace Admins can view and manage these pipelines. The only workaround I have found is to define every individual pipeline within Terraform to manage the ACLs, but this adds significant overhead.

As a small team, we simply need our Data Engineers to have visibility and control over all SDPs without making them full Admins. Does anyone know of a cleaner solution or a group-level permission setting that would solve this?

Re: Managing Spark Declarative Pipelines Permissions

Abeshek — Tue, 16 Dec 2025 05:05:42 GMT

This is a common governance challenge when teams move to Terraform-driven Databricks environments, trying to balance least-privilege access with operational efficiency. We see this frequently as platforms scale beyond a small admin group.

At Kanerika, as a Databricks partner, we help teams design role-based governance models that give data engineers the right level of visibility and control without expanding workspace admin access or increasing operational overhead.

If you’re open to it, happy to walk through how we typically approach this. Let me know a convenient date and time.

Re: Managing Spark Declarative Pipelines Permissions

nulltype — Wed, 17 Dec 2025 21:01:47 GMT

Our Solution: We moved job and pipeline permissions to DAB configuration files for streamlined enforcement. Terraform will remain the source of truth for workspace-level permissions only.