All Data Governance posts

Re: UI sends empty managed_identity_id, breaks storage credential creation with system-assigned Acce

camit001 — Fri, 31 Jul 2026 04:45:35 GMT

Thanks for sharing the detailed workaround! This is incredibly helpful.

Your step-by-step CLI approach makes it much easier to understand how to configure the storage credential and update the metastore when the UI doesn't expose the required option. The verification steps at the end are especially useful for confirming everything is configured correctly.

I'm sure this will save others a lot of time if they encounter the same issue. Really appreciate you taking the time to document the complete solution!

Re: Data contracts on Databricks: what does a working MVP actually look like?

honey_sharma — Thu, 30 Jul 2026 06:05:29 GMT

A practical MVP is to keep Bronze permissive for ingestion, but enforce contracts from Silver onward. Store the contract in Git, version it alongside pipeline code, validate it in CI using the Data Contract CLI, and publish approved changes with a simple changelog. For notifications, integrate contract diffs with Slack, Teams, or email so downstream consumers know before deployment. Ownership should ideally stay with the data product owner, with upstream teams reviewing contract changes as part of their release process.

Re: Data contracts on Databricks: what does a working MVP actually look like?

binlogreader — Thu, 30 Jul 2026 02:48:45 GMT

@emma_s @DoTA Thank you for your incredibly detailed notes! I am going to sit with them for a while to internalize, and then will respond back with some ideas or questions.

Re: Data contracts on Databricks: what does a working MVP actually look like?

DoTA — Wed, 29 Jul 2026 13:51:44 GMT

In practice, on large enterprise scale I found that applying data contracts is very tricky, and there are some major problem to it:

1. There are too many legacy systems, and aligning the owners of the system and the ability to alter the system alone consume too much energy (if the system even allow changes without going to a vendor and begging them for a CR which cause thousand of dollars). And, there are not only internal systems but also external systems (such as GA firebase, appsflyer as an example) which does not provide data contract OOTB.

2. If you manage to figure out part 1, applying data contract is not always straight forward, it really depends on the consumer. Some consumers want to stop the pipeline as soon as there is something wrong, some doesnt. Hence the general approach is to apply a pre_check (this is a task in the workflow) where you define the condition to allow the task to continue its transformation process. But it all boiled down to alignment and enterprise design, do not couple alert and stopping mechanism into a single pipeline as you will find your self aligning among different people with different risk appetite. Seperate them out of the golden layer (if you are following medallion architecture), put the hard stop in the consumption layer.

3. If you manage to get here, the alerting is easy, put it in the pre-check of the workflow it self. But beware, you will have thousands of alert everyday, so better to consolidate into a metadata table and send that table as a report daily.

Re: Enterprise Unity Catalog RBAC model in Databricks

DoTA — Wed, 29 Jul 2026 13:41:50 GMT

There are 2 aspects to this, it really depends on the enterprise architecture and data governance standards that your company needs to follow.

1. On the enterprise architecture, if your company is following medallion architecture, it is not wise to transfer the ownership back to the business owner, the SPP should still how accountable for the bronze, silver and golden layer. You should only transfer back the ownership on business layer (such as data marts if you have).

2. On the governance side, it really depends on the company context.

For some company such as mine, it is require for the Data Platform to be air-gapped as no privilage data (PII) is allow to be exported from the workspace, hence we are required to use advance feature to meet these requirement:

1. Such as the new feature exclusive group (b4 this it is seperate workspaces for sensitive and non-sensitive data).

2. Consideration of using ABAC.

Please ensure you look on both side before finalizing, as architectural decision doesnt make or break right away, it start breaking in the long run.

Re: View column comments

Pinei — Tue, 28 Jul 2026 18:23:46 GMT

Why are COMMENT ON statements and inline COMMENT clauses not supported for non-materialized views defined in Lakeflow Declarative Pipelines?

Databricks SQL supports view-level and column-level comments, but the same functionality does not appear to be available when non-materialized views are managed by a declarative pipeline. Is this a current product limitation, and is support planned?

Re: Data contracts on Databricks: what does a working MVP actually look like?

emma_s — Tue, 28 Jul 2026 15:18:50 GMT

Hi,

You've hit a problem that most data teams experience, some more than others! From my personal experience I'd be very wary about going down the data contract route. Unless you can get true ownership from the application team, which I've not yet seen. Usually, I've seen that it ends up as a sticking point to ingesting new datasets and what you normally end up testing with them is whether the contract is right rather than whether the actual dataset is right. But to answer your questions.

1. Usually the contract is stored as Yaml, in the data teams github repo, it could be put in a UC volume. It could work to have it in the App dev team's repo, but it if you have your data engineering patterns in a different repo this could cause issues.

2. I've seen it deployed in both the CI or the runtime. Never on the catalog. I think this would be challenging as there aren't many things that can actually be enforce on the table itself.

3. Usually, you would build the alerts into your CI or Pipelines and define who you would notify and this goes back to 1, who owns the contract? Do you want to be in a position where the appdev team changes something it breaks your pipeline and then you're reliant on them to fix it, which may not be a top prioirty for them.

4. I'm not convinced the two things can coexist together. I think on a table, you either need data contracts or schema evolution modes. My advice is to consider the purpose of the contract. When we had to make this situation better, we actually just added the need for a data engineering reviewer on any PRs that required a schema change. A lot of this, though, will depend on the culture and nature of the organisation you work for.

I hope this helps you think about it. As a side note, these are personal experiences rather than sanctioned Databricks best practices.

Many Thanks,

Emma

Re: Does user_identity in system.access.audit reflect the actual logged-in user for Service-Principa

emma_s — Tue, 28 Jul 2026 15:06:20 GMT

Hi, Everything @binlogreader has commented above is correct, the app service principal will be what appears in the system tables and if you'd like to get individual user authentication you'll need to use OBO authorisation, or pull somthing from the headers, but this would be seperate to the system tables. To give some additional pointers:

- The run as and run by are used for jobs and shared group compute where one identity triggers the work but it's run as another.

- You will be able to extract the user from the headers. Docs are here: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/http-headers

There are some nice examples in the docs of the usable headers.

Thanks,

Emma

Re: Enterprise Unity Catalog RBAC model in Databricks

Lu_Wang_ENB_DBX — Mon, 27 Jul 2026 15:43:00 GMT

Yes — this is a solid enterprise baseline. Your pattern matches Databricks guidance to automate UC with Terraform, use service principals for automation, and keep ownership / MANAGE narrowly assigned.

Single Terraform SP in a Metastore Admin group: Yes, acceptable. Using a service principal for IaC is recommended, and Databricks explicitly allows metastore admin to be assigned to a user, service principal, or group; group-based assignment is strongly recommended.
Would I keep that SP highly privileged forever? Only for central platform automation. Metastore admin is optional but highly privileged, so use it sparingly and prefer OAuth auth for the SP.
Storage credentials + external locations owned by Metastore Admin / platform group: Yes, that’s the cleaner pattern. These are top-level metastore objects, and Databricks recommends limiting creation/management of external locations to admins or trusted data engineers because they are broad, high-impact objects.
Catalog + schema ownership transferred to domain groups: Yes, good practice. Owners of catalogs and schemas can manage grants on child objects, and Databricks says catalog/schema admins should control downstream access while ownership should be assigned sparingly.
Small improvement: transfer ownership of external locations (and ideally storage credentials) from the creator SP to a platform admin group, not leave them implicitly on the SP.
RBAC roles: Your set is a good minimum: Metastore Admin, Catalog Owner, Schema Owner, Schema Write, Schema Read.
For larger enterprises, I’d usually add/clarify one more role: Platform Storage Admin for storage credentials/external locations, even if today it is the same people as Metastore Admin. That gives cleaner separation of duties as you scale.
Also remember Account Admin and Workspace Admin remain separate required admin layers outside your UC data roles.

Bottom line: Yes, good design. I would tweak: keep the single Terraform SP, but make its power come from a dedicated metastore-admin group, and make platform groups own storage credentials/external locations while domain groups own catalogs/schemas.

Re: Can I connect Fabric Data Agent With Databricks Genie One as external connection

Lu_Wang_ENB_DBX — Mon, 27 Jul 2026 15:37:08 GMT

This seems to be a duplicate question, please see my answer on the other thread.

Re: Pipeline still needs USE SCHEMA on an old schema it no longer writes to

binlogreader — Mon, 27 Jul 2026 13:04:16 GMT

@keshavmonga22 Is there anyone in your team with Admin access that can grant you temporary access or can check the event log?

Can I connect Fabric Data Agent With Databricks Genie One as external connection

RamenBhar — Mon, 27 Jul 2026 11:12:26 GMT

Hi

I have a fabric semantic model. I have created a fabric data agent (published) on top of the semantic model. Now i have created a unity catalog connection along with service. Is MCP connection is true. I can use this connection in AI playground or genie code to interact with the semantic model data. All is working. Region is uksouth.

But is there any way i can use this connection as data source in genie one? I have checked the documentation but cant see this connection in the more connection list. is this because of the region

Enterprise Unity Catalog RBAC model in Databricks

APJESK — Mon, 27 Jul 2026 07:57:18 GMT

I'm designing an enterprise Unity Catalog RBAC model in Databricks and would like feedback on whether this follows best practices.

My current design is:

A single Service Principal is used by Terraform to provision all Unity Catalog objects.
The Service Principal is a member of the Metastore Admin group.
Using Terraform, the Service Principal:
- Creates the Storage Credentials and External Locations.
- Creates the Catalog, then transfers ownership to the appropriate Catalog Owner group.
- Creates the Schema, then transfers ownership to the appropriate Schema Owner group.
The Service Principal remains the automation identity, while business ownership is transferred to the respective owner groups.

My RBAC roles are:

Metastore Admin
Catalog Owner
Schema Owner
Schema Write
Schema Read

Is this considered a good enterprise-scale design and aligned with Databricks Unity Catalog best practices?

Specifically:

Is it a best practice to use a single Terraform Service Principal that belongs to the Metastore Admin group for provisioning?
Should the Metastore Admin group continue to own Storage Credentials and External Locations, while Catalog and Schema ownership is transferred to the respective owner groups?
Are these RBAC roles sufficient for a large enterprise, or would you recommend adding or changing any roles?

Re: Does user_identity in system.access.audit reflect the actual logged-in user for Service-Principa

tetsuro — Mon, 27 Jul 2026 07:48:02 GMT

@binlogreader

Thank you very much for the clear and detailed explanation.
This clarifies the distinction between pure SP mode and OBO.

My understanding is that, in pure SP mode without user_api_scopes, all Databricks actions initiated by the app are audited as the service principal. Therefore, the actual logged-in user cannot be identified from system.access.audit alone. In this case, acting_resource remains NULL, and run_by / run_as should not be treated as fields that provide the end-user identity for the Databricks Apps SP-mode scenario.

I also understand that the app can obtain the logged-in user identity from headers such as X-Forwarded-Email and record it in application-owned logs. However, if individual user attribution is required in the platform-owned audit trail, OBO is the appropriate mechanism.

We will perform a controlled test with a known user action and verify the actual records in system.access.audit. Thank you again for the practical guidance.

Re: Pipeline still needs USE SCHEMA on an old schema it no longer writes to

keshavmonga22 — Mon, 27 Jul 2026 07:11:09 GMT

Hi @binlogreader and @adnan_alvee , thank you for the replies. But I don't have access to system.access either.

We had temporarily granted USE SCHEMA permissions on the older schema to run the pipelines.

Since then, I have deleted the schema entirely since it was anyways inactive now. Please note that when we had changed schemas, we had recreated the tables from scratch in the new schema.

Re: Does user_identity in system.access.audit reflect the actual logged-in user for Service-Principa

binlogreader — Mon, 27 Jul 2026 02:03:34 GMT

@tetsuro With app authorization only, the end user does not appear in `system.access.audit` for the calls your app makes. The sample query you found is showing on-behalf-of behavior specifically, and the query itself says, the CTE is named `obo_events`, and it filters on `identity_metadata.acting_resource IS NOT NULL`, a field that only populates when the app acts with a user's forwarded token.

In pure SP mode, every call your app makes authenticates with the SP's client ID and secret, no user identity reaches Databricks, so the audit row shows the SP's application ID in `user_identity.email` and `acting_resource` stays null.

On question 2: `run_by` and `run_as` are documented for contexts where one identity triggers work that executes as another, such as jobs and dedicated group compute. Nothing in the audit log reference ties them to Databricks Apps.

Here are few things to try,

1. Have one user perform a known action in the app, then query `system.access.audit` for that window. Seeing the SP's application ID and a null `acting_resource` gives your compliance file evidence rather than a doc interpretation.

2. Every request forwarded to your app carries the logged-in user's identity in headers (`X-Forwarded-Email`, `X-Forwarded-Preferred-Username`, per the "Access HTTP headers passed to Databricks apps" docs page), so the app can record which user drove which action.

3. If compliance requires the user in the platform-owned audit trail itself, OBO is the mechanism built for that. Those calls then audit as the user with your app in `acting_resource`, which is exactly the rows the sample query surfaces.

Does user_identity in system.access.audit reflect the actual logged-in user for Service-Principal-mo

tetsuro — Mon, 27 Jul 2026 00:25:19 GMT

We're deploying a Databricks App using Service Principal (SP) mode (not using OBO's user_api_scopes).

The sample query "List app actions performed on behalf of users" in Logging and Monitoring for Databricks Apps shows system.access.audit.user_identity.email commented as "Logged-in user" (the actual person who logged in), with identity_metadata.acting_resource holding the app/SP-side identifier.

My questions:

1. Does this behavior only apply when On-Behalf-Of User Authorization (OBO) is enabled? Or does a pure SP-mode app (without user_api_scopes) also populate user_identity.email with the actual end user's email whenever they interact with the app?
2. The Audit log system table reference documents run_by / run_as fields under identity_metadata — do these also apply to the Databricks Apps SP-mode case specifically?

We need to confirm, for internal compliance purposes, whether individual end users can be identified at the audit-log level, and couldn't find an explicit statement in the official docs covering this exact combination. Any pointers to documentation or real-world behavior would be appreciated.

Data contracts on Databricks: what does a working MVP actually look like?

binlogreader — Sat, 25 Jul 2026 20:03:46 GMT

Hi All, I'm trying to solve two related problems on our lakehouse and I keep arriving at data contracts as the answer, but I can't find much on what a first working version looks like in practice on Databricks:

(a) Upstream application teams change their database schemas without telling us. We ingest their tables through CDC, and schema evolution keeps the pipelines alive (new columns get added or land in the rescued data column), but that is absorption, not communication. We find out after the change is already flowing, and purely semantic changes, like a column's meaning or allowed values shifting, never trip anything at all.

(b) When a change reaches our silver and gold tables, consumers find out when their dashboards or queries break. Unity Catalog lineage tells them where data came from, but nothing tells them ahead of time that something is about to change.

Here is the the closest end-to-end example on building a data product with Databricks. A datacontract.yaml per data product, the Data Contract CLI to generate DDL and run acceptance tests against the actual Unity Catalog table, deployment through Asset Bundles, tests running as job tasks, and metadata published to a registry so consumers can discover the product. It's a good picture, but it's a demo of one product, and it leaves my main questions open.

While thinking about what an MVP may look like, here are some questions that I request some guidance on:

1. Where does the contract live, and who owns it? For the upstream problem, the contract only helps if the application team's CI fails when their migration breaks it. Has anyone actually gotten an application team to adopt a contract for tables they consider their internal database?
2. Where do you enforce? Options seem to be CI (datacontract test on a schedule or on deploy), runtime (DLT expectations generated from the contract), or the catalog itself (constraints, tags). What combination is worth the effort first
3. How do downstream consumers actually get notified? A registry is discovery, not notification. Is anyone driving alerts or a changelog from contract diffs, and how?
4. Is anyone running the Data Contract CLI against Unity Catalog in production? How does it coexist with schema evolution modes that are deliberately permissive at bronze?

Appreciate your time!

Re: CONNECTION Permission Issue - Can't use existing connection for Community Custom Connector

balajij8 — Fri, 24 Jul 2026 18:46:56 GMT

Hi ChristianRRL,

Most likely a classic UI race condition or state management glitch with the UI which is common as Community Custom Connectors are in beta and under active development. Interfaces and behavior are subject to change.

What's likely happening here is that the initial page load is failing to fetch your existing connection metadata before rendering the component. Because the UI doesn't see any existing connections in that split second, it defaults to the fallback state of prompting you to create a new one. That immediately trips the CREATE_CONNECTION permission error since your role is properly restricted to USE CONNECTION.

Refreshing the page forces a clean state reload, allowing the backend API call to resolve properly and populate the drop-down. Your USE CONNECTION grant is perfectly fine, the workspace UI is just getting ahead of itself on the initial load.

You can document the behavior for your team and file a bug here. If you are rolling this pipeline out to a production environment, keep an view on the behavior during updates as frontend caching and routing bugs like this are usually patched silently on the road to General Availability. You can try using Managed connectors if feasible.

Re: CONNECTION Permission Issue - Can't use existing connection for Community Custom Connector

ChristianRRL — Fri, 24 Jul 2026 14:38:36 GMT

+Additional context, following the same instructions on my ADM account. I do have the ability to "Create connection", but once again I can't find existing connections.

This is odd because (A) I should be able to leverage existing connections and not need to create a new one every time, and (B) both my ADM and regular accounts have the same access as far as I can tell.

... Wait, this is EXTRA WEIRD (writing in real time). I see my existing connections now after refreshing the page!

... Likewise, I went through the same steps on my non-ADM account. Originally, it came up just like I described originally (no existing connections came up). But after I refreshed the page, IT CAME UP!

Not even sure what to call this, but this seems like a genuine bug.