All Data Governance posts

Re: Personal Compute Policies

ivanvyd — Mon, 21 Sep 2026 15:31:23 GMT

@Ankitkalra40 since Serverless is disabled, there is one newer native option worth checking: as of the September 17 release, base environments are supported on classic compute in Beta.

I haven't tried it yet, honestly, but as you might want to consider it as an alternative to other answers.

On DBR 19+ with Standard access mode, you can use Dependency mode = Environments and provide admin-managed workspace base environments.

One caveat for the InfoSec requirement: this is not a hard allowlist by itself, because users can still add notebook-level dependencies. If only approved packages must be installable, I'd keep the trust boundary at the package source: enforced compute-policy libraries / approved wheels in UC Volumes, plus a curated private package repository and egress restrictions.

Also worth noting that Environments mode currently targets interactive all-purpose notebooks; classic jobs don't use it yet.

Re: Personal Compute Policies

balajij8 — Mon, 21 Sep 2026 13:12:53 GMT

@Ankitkalra40

You can handle it natively using Compute Policies with Enforced Libraries instead of building a custom DBX app to sync libraries. Workspace admins can build per team compute policies that enforce specific library versions. The key is that users on these policies are completely locked out of installing or uninstalling libraries on this compute. More details here

You set it up directly in the policy's Libraries tab that accepts up to 500 libraries per policy. Instead of using a monolithic init script, you can create dedicated policies per team / dependency matrix eg., one policy mapped to Engineers and another for Scientists

You can pair these policies with Unity Catalog Volumes to address the InfoSec approval process. You can store the InfoSec-validated .whl, .jar and requirements.txt files in a UC Volume and use Unity Catalog grants to restrict READ access to the appropriate teams. More details here

Compute policies can point directly to these volume paths instead of public repositories. It creates a tightly gated, native workflow - InfoSec reviews a library and drops the package into the secure volume, the team's compute policy picks it up and the engineers cannot deviate.

Re: Personal Compute Policies

Ankitkalra40 — Mon, 21 Sep 2026 11:27:30 GMT

Hi Thomaz,
I agree with most of your recommendations which are in line with the best practices as well.
However, the client is a bit strict about their data policies and Serverless is disabled at their org level.

This is the primary reason a lot of above can not be directly implemented or tested for their requirement.

Is there any other alternative that you can help with? Happy to connect separately on this as well

Re: Personal Compute Policies

ThomazNeto — Mon, 21 Sep 2026 11:12:57 GMT

Hi Ankit,

Before building anything custom, have a look at three things that already exist in the platform. Together they cover most of what you described.

Workspace base environments. A workspace admin defines a YAML with the serverless environment version and a pinned list of Python dependencies (requirements file, wheels in a UC volume, an internal index-url). Databricks pre-builds and caches it, users pick it from the Environment panel, and admins can star one as the workspace default. You're limited to 10 per workspace, so think "approved profiles" (data-eng, ml-sklearn, ml-mlflow) rather than one per person. Base environments also work on classic compute through "manage dependencies using environments". This is probably the closest thing to what you're prototyping, without the app and the Lakebase sync.
https://docs.databricks.com/aws/en/admin/workspace-settings/base-environment
https://docs.databricks.com/aws/en/compute/serverless/dependencies
Compute policies with libraries. For classic compute, a policy can carry up to 500 libraries (PyPI, wheels in volumes, requirements.txt) that get installed automatically, and the docs are explicit about the side effect you want: "Users can't install or uninstall compute-scoped libraries on compute that use this policy." Databricks also recommends policies over init scripts for library installs. Note this blocks compute-scoped libraries; notebook-scoped %pip still works unless you cut off the index (see next point).
https://docs.databricks.com/aws/en/admin/clusters/policies
Control the source, not the list. The docs say private mirrors like Nexus or Artifactory are supported via --index-url in %pip and in base environment YAML, and admins can set a private repo as the default pip source for serverless. If InfoSec curates what's in the mirror, the approval happens once, at the repository, and every install path inherits it. That's cleaner than approving versions inside Databricks.
https://docs.databricks.com/aws/en/libraries/

One more, since you tagged Unity Catalog: on standard access mode, JARs, Maven coordinates and init scripts need to be in the UC artifact allowlist (MANAGE ALLOWLIST privilege). It doesn't cover PyPI, so for Python the mirror is the control point.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/allowlist

Your app might still make sense as the request/approval UI, but I'd have it write base environments and policy libraries rather than manage installs itself.

Hope this helps.

Re: Changing Root Metastore Location

Ashwin_DSA — Sat, 19 Sep 2026 17:17:33 GMT

Hi @yawningbrain,

You can't repoint the metastore root in place, so the supported path is to give every catalog its own managed storage on the new ADLS and then remove the metastore root. Here is the sequence with the relevant docs.

Step 1 - Give your business catalogs explicit managed storage on the new ADLS. First, find which catalogs still fall back to the metastore root. Run "DESCRIBE CATALOG EXTENDED <catalog_name>" and look at the Storage Root field. Any catalog with no catalog-level location is relying on the root.

For each of those, register an external location for the new ADLS, backed by a storage credential with an Azure managed identity, then set the catalog location.

ALTER CATALOG my_catalog SET MANAGED LOCATION 'abfss://my-container@mynewstorage.dfs.core.windows.net/my_catalog';

This needs Databricks SQL or Databricks Runtime 18.1 and above, and you need CREATE MANAGED STORAGE on the external location plus ownership or the MANAGE privilege on the catalog. The managed location string must be 150 characters or fewer. You can refer to these links. ALTER CATALOG and Specify a managed storage location.

Altering the managed location only affects new managed tables and volumes created after the change. Existing managed objects stay where they are and keep working, so your pipelines will not break. Worth knowing too that storage precedence runs schema location over catalog location over metastore location, so removing the root only touches catalogs that have no catalog-level location.

Step 2 - Remove the metastore root. Once your business catalogs have explicit storage, go to the account console, open Catalog, select your metastore, open the Configuration tab, and click Remove next to the ADLS Gen 2 path. What happens next is documented here. Catalogs that already have their own root are unaffected. Catalogs without one are given the old metastore root path as their catalog-level managed storage, which the docs call push-down, and access continues without interruption. Depending on how your metastore was created, a new external location named prior_metastore_root_location and an associated storage credential may be auto-created for the old path. After this, every new catalog must specify its own location.

If any catalog must keep using the exact old path after removal, you can pre-register that path as an external location and set it explicitly before you remove the root. That gives you a named external location and credential you control instead of the auto-generated prior_metastore_root_location.

Step 3 - The system and __databricks_internal catalogs. These two need separate consideration, and the removal doc above does not actually describe either of them. It only covers the generic push-down rule for ordinary catalogs, so treat these as follows.

system holds your account's operational system-table data, and that data is stored in a Databricks-hosted storage account in the same region as your metastore, shared with you through open sharing. It does not physically live in your metastore root. So even if a legacy path shows up in catalog properties after push-down, that alone does not mean the system data sits in the old account, and retiring the old ADLS should not break system tables.

__databricks_internal is a platform-managed catalog used for pipeline materialisations such as materialised views and streaming tables, and Databricks advises against accessing its tables directly. Unlike system, this data can genuinely reside in customer-managed storage, so this is the one that could still tie you to the legacy account. The removal behaviour for it is not documented, so I would have Databricks Support confirm the migration or cleanup procedure before changing any internal paths or retiring the storage account.

Removing the root does not get you off the legacy ADLS. Push-down only rewrites the pointer. It moves no data. So root removal is safe and low-impact, but it is not the step that lets you decommission the legacy container.

Before assuming zero dependency, confirm each business catalog actually has its data on the new ADLS. If a business table was a managed table under the old root, its files may still sit in the old container even though the table reads fine. The thing to check is whether you have any Unity Catalog pipelines, materialised views, or streaming tables, since those materialisations land in __databricks_internal and are the tangible remaining tie to the old storage. For a full cutover later, deep-clone or CTAS the remaining managed tables into the new locations. Keep the old account and its access permissions intact until those dependencies are resolved, because the documented root-removal behaviour preserves read access through a repointed pointer, it is not a guarantee that the underlying storage is safe to delete.

This should ideally be zero-impact for running pipelines. Existing managed tables keep their data in the original storage location regardless of changes to the metastore or catalog root. Only newly created managed tables and volumes use the updated location. External tables are unaffected since they reference their own storage paths.

If you use open sharing and the metastore root was used as shared notebook storage, you must remove the notebook from the share and re-add it using a dedicated storage location before you can remove the root.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Unity Catalog-Error Creating Table ErrorClass=INVALID_STATE] Failed to access cloud storage: Abf

dvhdcrd — Sat, 19 Sep 2026 00:15:35 GMT

I just faced same issue and was navigating to this thread, and I proactively enabled this option in the metastore settings:

that resolved the issue.

Hope this helps someone.

Re: Feature enablement for Foundation Model Unity Catalog permissions

tom_n — Fri, 18 Sep 2026 12:57:05 GMT

Hi, I have contacted the account team associated to your account to get in contact.

Personal Compute Policies

Ankitkalra40 — Fri, 18 Sep 2026 06:31:52 GMT

I am working with a client to set up their budget policies and compute governance.

They have a team of Engineers and Scientists that work across different versions of various libraries like python, pandas, numpy, scikit, mlflow etc.

Since each user requires diff versions, having one init script defeats the purpose. Also, any library and the version to be installed is supposed to be validated by InfoSec.

I have proposed a rather niche solution to this that utilizes DBX app as the interface and syncs the approved libraries via Lakebase instance.

We are testing this out but wanted to check if there is something easier available within Databricks at a platform level.

Thanks in advance!

Re: Feature enablement for Foundation Model Unity Catalog permissions

broccobroccolis — Thu, 17 Sep 2026 02:51:58 GMT

Thank you @tom_n.

There is no "Foundation Model Permissions" listed under Previews in my Account Console, could you please help to enable it for my account?
Account ID: 844514a9-a1ed-484a-9baa-6699d5599bd6
Could you also assist me with the legacy endpoints:
1. To close them without removing them - is there any documentation reference to disable those specific Foundation Model API endpoints for workspaces or to migrate that traffic onto Unity Gateway where it's UC-governed?
2. To remove those endpoints entirely - there is no Enforce Unity Gateway in my workspace settings. Could you help confirm if this is unavailable in my region? Otherwise we will need to go down the route of disabling the endpoints.

Re: Feature enablement for Foundation Model Unity Catalog permissions

tom_n — Tue, 15 Sep 2026 13:24:05 GMT

It persists after enablement. The legacy databricks-* endpoints are a separate serving path from both pay-per-token and provisioned throughput, so they aren't gated by the system.ai EXECUTE revoke. It isn't pre-enablement behaviour that clears once the feature is on.

The permissions page doesn't cover this case, so I've confirmed it on our side.

So the account needs both:

1. Enable Foundation model Unity Catalog permissions. This enforces the revoke on the pay-per-token and AI Functions paths automatically, and covers new models going forward.
2. Clear the endpoints that enablement doesn't touch: delete any disallowed provisioned-throughput endpoints, and remove the legacy databricks-* endpoints. That legacy removal is handled on our side, either via the Enforce Unity Gateway workspace setting where it's available in your region, or a backend request we raise to disable the specific Foundation Model API endpoints. I'll confirm which applies to your workspaces and set it up with you.

Quickest way to scope it: once the feature is enabled, re-run the same call with EXECUTE still revoked. If it's blocked, the only remaining work is clearing the standing endpoints. If it still succeeds, it's routing to an existing endpoint rather than the pay-per-token path.

Re: Feature enablement for Foundation Model Unity Catalog permissions

ivanvyd — Tue, 15 Sep 2026 10:36:46 GMT

Thank you, @tom_n. Could you please clarify one point against the permissions documentation?

It says pay-per-token calls enforce the restrictions automatically after enablement, while provisioned-throughput endpoints require manual removal.

Does the legacy databricks-* exception still apply after Foundation model Unity Catalog permissions is enabled? Or does it describe the behavior before enablement? That would help establish whether this account needs the feature enabled, additional endpoint restrictions, or both.

Re: Feature enablement for Foundation Model Unity Catalog permissions

tom_n — Tue, 15 Sep 2026 09:23:07 GMT

The reason the two calls behave differently is that databricks-* and system.ai.* are separate serving and authorisation paths, and only the system.ai.* path runs through Unity Catalog.

- system.ai.claude-sonnet-4-5 resolves to a UC model service, so your EXECUTE revoke applies and the call fails as expected.
- databricks-claude-sonnet-4-5 resolves to the legacy default Foundation Model API serving endpoint. Direct calls to it don't pass through UC model-service authorisation, so revoking EXECUTE on system.ai never touches it.

The feature and how to enable it

In the Account Console it's listed under Previews as "Foundation Model Permissions" (account-level). An account admin toggles it on there and it takes effect immediately. If the toggle isn't visible, we can get it enabled for you from our side. Once it's on, revoking EXECUTE on system.ai is enforced across the UC-governed paths: ai_query, the AI Playground, and the newer gateway endpoints.

One thing to flag: enabling the feature covers newly created model services automatically, but it doesn't retro-fit services that already exist. For those you'll need to revoke EXECUTE per model service. New models landing in system.ai also grant EXECUTE to all users by default.

Enforcing consistently across all paths

1. Revoke EXECUTE at the catalog, schema and securable level (all three), scriptable through the UC Grants API.
2. Enable Foundation Model Permissions so new models don't quietly re-open access.
3. Allow-list the models you do want, either by re-granting EXECUTE per approved model service, or with an ABAC grant policy (grant everything except models carrying a "blocked" tag).
4. The legacy default databricks-* endpoints are the one path UC can't reach. To close them fully, speak to your account team (if you have one) to disable those specific Foundation Model API endpoints for your workspaces, or you migrate that traffic onto Unity Gateway where it's UC-governed.

Does serverless compute bypass Unity Catalog?

No. Serverless SQL and serverless compute enforce Unity Catalog in full. Serverless didn't ignore UC here: your ai_query("databricks-…") call routed to the legacy serving endpoint, which sits outside UC authorisation. Enabling Foundation Model Permissions and dealing with the legacy endpoints closes that gap.

Reference: Foundation model Unity Catalog permissions — docs.databricks.com/machine-learning/foundation-model-apis/model-uc-permissions

Feature enablement for Foundation Model Unity Catalog permissions

broccobroccolis — Tue, 15 Sep 2026 03:43:09 GMT

I am trying to restrict workspace users' access to Databricks Foundation Models using the guidance in the Foundation Model Unity Catalog Permissions documentation.

I have revoked EXECUTE permission for all users from the system.ai schema. However, workspace users can still successfully query foundation models via ai_query() using the databricks-* serving endpoint names:

-- This SUCCEEDS (should be blocked)
SELECT ai_query("databricks-claude-sonnet-4-5", "whats 1+1?") AS response

-- This FAILS with RESOURCE_DOES_NOT_EXIST
SELECT ai_query("system.ai.claude-sonnet-4-5", "whats 1+1?") AS response

The system.ai.* path returns a RESOURCE_DOES_NOT_EXIST error (which I assume is expected) after revoking EXECUTE. However, calling the same model via the databricks-* endpoint name bypasses the UC permission check entirely.

The documentation states: "Foundation model Unity Catalog permissions is generally available but requires enablement. Reach out to your Databricks account team to enable this feature."

What is the Foundation Model Unity Catalog Permissions feature called in Account Previews and how can it be enabled? So that the EXECUTE revoke on system.ai is enforced consistently across all access paths — including pay-per-token databricks-* endpoints called via ai_query().

Side question: does serverless compute bypass Unity Catalog permissions?

Re: Changing Root Metastore Location

ivanvyd — Sat, 12 Sep 2026 01:45:09 GMT

@yawningbrain this is a great topic. The Azure Databricks documentation⁠ supports removing the existing metastore root and adding a new path. You cannot edit the existing path in place. This does not migrate existing data.

When you remove the root, catalogs without their own storage location inherit the old path as their catalog-level storage. That preserves access, but means replacing the metastore root does not necessarily remove dependencies on the legacy account.

The two catalogs you mentioned need separate consideration:

system: Operational system-table data is stored in a Databricks-hosted storage account, according to the system tables reference⁠. A legacy path displayed in catalog properties would not, by itself, establish that those tables are physically stored there.
__databricks_internal: Databricks identifies this as an internal catalog used for pipeline materializations and advises against accessing its tables directly. And I never tried to. For that dependency, I would get Databricks Support to confirm the migration or cleanup procedure before changing internal paths or retiring the storage account.

That said, keep the old account and its access permissions intact until those dependencies are resolved - the documented root-removal behavior is not a guarantee of zero impact when the underlying storage is deleted.

Are you seeing actual objects stored in the legacy account, or only the old path in these catalogs' properties?

Changing Root Metastore Location

yawningbrain — Fri, 11 Sep 2026 14:49:17 GMT

We currently have a default root metastore location linked to an ADLS container.
At the same time we are working on a ADLS migration project.
All of the tables and files (everything we use for day to day business related) have moved to the new ADLS so there are no dependencies on the legacy ADLS - except for the system and __databricks_internal catalogs.

How can we safely change the root metastore location without breaking our environment - ideally with minimal to 0 impact on our pipelines and system metadata.

Re: Harmonizing Informatica Governance Policies Across Databricks and Power BI Import Mode

ivanvyd — Wed, 09 Sep 2026 09:55:48 GMT

@fjrodriguez the key is to keep one policy source while accepting two enforcement points. Informatica should own the business intent and approval; Unity Catalog and Power BI should enforce platform-specific artifacts generated from the same versioned policy contract.

For Unity Catalog, use a platform-owned pipeline to translate that contract into governed tags and ABAC policies. Grant the service principal narrowly scoped privileges—for example, ASSIGN and APPLY TAG, plus scope-level MANAGE and EXECUTE on any policy UDF—rather than metastore ownership.

“Databricks recommends ABAC policies when you need consistent row filtering and column masking across many tables.” Databricks documentation

For AI/BI dashboards, use Individual data permissions when policies must be evaluated for each viewer; shared permissions execute as the publisher. Genie applies Unity Catalog row filters and column masks per user.

Power BI Import mode cannot pass the report consumer’s identity back to Unity Catalog. You therefore cannot remove Power BI-side enforcement, but you can remove manual policy re-authoring: generate a dynamic RLS entitlement table keyed by USERPRINCIPALNAME() from the same policy contract. For column restrictions, use OLS, exclude sensitive columns, or import a pre-masked projection. Keep consumers in Viewer or app roles because RLS does not restrict workspace Admin, Member, or Contributor roles.

For audit evidence, retain the policy version, effective date, deployment manifest, and test results. Compare SHOW EFFECTIVE POLICIES, system.information_schema.abac_policy_definitions, and system.access.audit with the deployed Power BI model roles. Run identical positive and negative test identities against both platforms. Deployment records show what was configured; identity-based tests demonstrate equivalent enforcement.

Harmonizing Informatica Governance Policies Across Databricks and Power BI Import Mode

fjrodriguez — Tue, 08 Sep 2026 17:11:01 GMT

What are the recommended architectures or best practices for harmonizing and enforcing governance policies (such as data masking and access controls) defined in Informatica across both Databricks and Power BI datasets running in Import mode?

What I can't find a clean answer for is how to keep those rules consistent across the stack:

The rule is decided and approved in the governance catalog Informatica, by the business.
It has to be enforced in Unity Catalog, so it applies to SQL, notebooks, dashboards and Genie.
And it has to apply in Power BI, where our semantic models are Import mode — thousands of users, so DirectQuery isn't viable for us on cost and latency. Which means row filters and column masks defined in Unity Catalog don't reach the report consumer.

So today the same business rule risks being expressed more than once, by different teams, with no way to prove they say the same thing on a given date

My questions to anyone who has been through this:

How do you get a policy that is authored in an external governance tool to land in Unity Catalog without that tool taking over permissions in your metastore?
With Power BI in Import mode, how do you avoid re-implementing the same restriction in DAX? Is there a pattern better than maintaining a user-level entitlement table?
How do you demonstrate to audit that both platforms are applying the same rule, rather than asserting it?
And more generally: where do you draw the line between the governance catalog and Unity Catalog — who defines, who enforces?

Re: Semantics as Code: Looking for Collaborators on Open Enterprise Business Semantics

vkondepati — Wed, 02 Sep 2026 20:52:48 GMT

We have introduced a new role, Semantic Steward, who is responsible for working closely with the Data Steward, Data Architect, and Platform Administrator to review and approve KPI definitions, as well as changes to existing KPIs.

A Governance Committee has also been established to ensure that KPI changes are properly controlled, reviewed, and approved, and that new KPIs are introduced based on business needs. The committee consists of the Semantic Steward, Data Steward, Data Architect, and Platform Administrator.

All changes to KPI definitions and metrics will be managed, reviewed, version-controlled, and auditable through a GitHub repository.

The metadata management tool will provide comprehensive information for each KPI, including KPI definitions, source data, metrics, measures, data models, business glossary terms, and data lineage. This will provide transparency into how each KPI is defined, calculated, and derived from underlying data sources.

Data classification tags will be used to control access to KPIs based on their sensitivity. Access requests for Confidential and Highly Confidential KPIs will require approval from the Semantic Steward in conjunction with the Data Steward.

Re: Semantics as Code: Looking for Collaborators on Open Enterprise Business Semantics

kartikchoudhary — Tue, 01 Sep 2026 06:29:28 GMT

This is an interesting approach, especially with AI agents becoming more involved in enterprise data workflows.

I think one of the key challenges will be keeping semantic definitions consistent as business requirements and underlying data models evolve. How are you thinking about governance and ownership of these definitions — for example, who approves changes to a metric or business entity definition, and how are those changes propagated across downstream platforms?

The connection between Semantics as Code, Unity Catalog, and AI agents is particularly interesting.

Semantics as Code: Looking for Collaborators on Open Enterprise Business Semantics

vkondepati — Mon, 31 Aug 2026 16:31:03 GMT

Infrastructure became code.

Pipelines became code.

Policies became code.

Why is enterprise business meaning still scattered across catalogs, BI models, spreadsheets, wikis, and tribal knowledge?

I’ve been working on Semantics as Code, an open-source, vendor-neutral approach for defining enterprise business meaning as version-controlled, testable, and deployable artifacts.

The idea is to treat business semantics with the same engineering discipline we apply to infrastructure and data pipelines.

Business entities, metrics, relationships, glossary terms, ownership, governance metadata, quality expectations, and AI context can be defined as code, validated through CI/CD, reviewed through Git, and generated for downstream data and AI platforms.

For example, instead of allowing every dashboard, data product, or AI agent to independently interpret what Revenue, Customer, or Active Customer means, we can establish governed semantic definitions that are reusable across the enterprise.

Why this becomes especially important with AI agents

As enterprises adopt agentic AI, I believe we face a problem beyond traditional data governance.

Data governance can answer:

“Can this agent access this data?”

Policy governance can answer:

“Is this agent allowed to perform this action?”

But we also need to answer:

“What does this business concept mean, and which definition should this agent use in this context?”

I’m exploring this as Meaning Governance—using Semantics as Code as a foundation for providing authoritative, governed business context to AI agents.

The open project currently supports semantic definitions for entities, metrics, relationships, glossary terms, quality expectations, governance metadata, and AI context. The reference implementation also includes generation targets for platforms and technologies including Databricks Metric Views, dbt, OpenMetadata, knowledge graphs, and AI context artifacts.

Looking for collaborators

I’d love to collaborate with people in the Databricks community interested in:

Databricks Metric Views and semantic layers

Unity Catalog and data governance

Agentic AI / AI agents

Enterprise metadata and business glossaries

Semantic models and knowledge graphs

Semantic interoperability

Data contracts and data quality

Governance-as-code / policy-as-code

Building Databricks adapters and real-world examples

I’m particularly interested in exploring how Semantics as Code + Databricks + AI agents could work together to create a governed semantic foundation where business meaning is portable, testable, traceable, and consumable by both humans and autonomous agents.

The project is open source, and contributions, architectural feedback, use cases, criticism, and research collaboration are all welcome.

Project documentation:

https://vkondepati.github.io/semantics-as-code/

GitHub:

https://github.com/vkondepati/semantics-as-code

If this problem resonates with you, I’d love to connect and collaborate.

Define once. Govern everywhere.