Community Articles topics

Row-level security for a RAG agent: what Unity Catalog enforces, and what you have to build

SvenRelijveld — Wed, 30 Sep 2026 17:19:43 GMT

Two colleagues ask the same chatbot the same question. Should they get the same answer, or

different ones because they are allowed to see different things? Every organisation that puts an AI

assistant on top of its own documents runs into this sooner or later.

Unity Catalog has a good implementation for that on calls to tables. Attach a row filter and a

column mask, and every reader gets their own view of the same object, evaluated against whoever is

asking. It does not have a type of grant for row filters *inside* a vector index. An AI Search

index is a Unity Catalog object with object-level grants, so you can allow or deny querying it, but

it has no row filters and no column masks. Filtering an index is a parameter you pass in the query,

from your own application code.

The 32nd Column: Why Your Z-Order or Clustering Key May Be Doing Nothing

Islam_hoti — Wed, 30 Sep 2026 13:39:18 GMT

You cluster a table on a column, run the query, and nothing improves. The layout command completed successfully. The column is definitely in the filter. The file count in the query profile has not moved.

Before assuming the clustering did not work, check whether statistics exist for that column at all. In a wide table they may not, and nothing in the process will tell you.

How skipping actually works

Delta records statistics for each data file: the minimum value, the maximum value, and the null count per column, plus the record count. At query time the engine compares your predicate against those numbers and discards files that cannot possibly contain a match. A file is never opened, so the cost of that file drops to zero.

Clustering exists to make those ranges narrow. If a file spans the entire range of a column, the engine cannot rule it out. If the file covers a small slice, most queries can discard it immediately.

The whole mechanism rests on one assumption: that statistics exist for the column you are filtering on.

Where that assumption breaks

For Unity Catalog external tables, statistics are collected on the first 32 columns defined in your schema. Not the 32 most useful columns. Not the columns you filter on. The first 32, in declaration order.

Column 33 onward has no statistics. A filter on one of those columns cannot skip a single file. The query returns correct results, runs at full scan cost, and reports no error.

This is easy to walk into. Wide tables grow by accretion. Someone adds a field, then another, then a nested struct whose fields also count toward the limit. Two years later a new filter column sits at position 41 and the table quietly stops being skippable on it.

The documentation is explicit on the consequence for Z-order: Databricks recommends not using ZORDER BY on columns that do not have statistics collected, because it is ineffective and consumes compute for nothing. Liquid clustering reads the same per-file statistics, so the same reasoning applies to a clustering key that sits past the boundary.

One important exception

For Unity Catalog managed tables, predictive optimization does not use a fixed 32-column list. It runs ANALYZE and collects skipping statistics on the columns that actually appear most often in your query filters, so the limit does not apply.

This matters because it means two tables in the same workspace can behave completely differently, and the difference is invisible in the query text. Before debugging, establish which regime your table is in. If it is a managed table with predictive optimization enabled, the column position is not your problem and you should look elsewhere.

How to check

Start with what the table is configured to collect:

SHOW TBLPROPERTIES my_catalog.my_schema.my_table;

Look for delta.dataSkippingStatsColumns and delta.dataSkippingNumIndexedCols. If neither is set and the table is external, you are on the 32-column default.

To see what was actually written rather than what is configured, read the statistics out of the transaction log. Get the path first, then read the add actions:

DESCRIBE DETAIL my_catalog.my_schema.my_table;

Take the location value, then:

SELECT add.stats FROM json.<location>/_delta_log/*.json WHERE add IS NOT NULL LIMIT 5;

The stats field is a JSON document with minValues, maxValues and nullCount objects. The column you care about either appears in there or it does not, and that single check settles the question.

The most direct behavioural evidence is in the query profile. Compare files pruned against files read for a filter on the suspect column, then run the same shape of query filtered on a column you know is early in the schema. If one prunes and the other does not, you have your answer.

Three ways to fix it

Name the columns explicitly. On DBR 13.3 LTS and above this is the cleanest option, because it decouples statistics from schema position entirely:

ALTER TABLE my_table SET TBLPROPERTIES ('delta.dataSkippingStatsColumns' = 'region, order_date, promo_code');

This property supersedes the numeric limit. Be deliberate about the list, since statistics are not free to compute or store.

Raise the count. Works on all runtimes, but remains order dependent, so it is a blunter instrument:

ALTER TABLE my_table SET TBLPROPERTIES ('delta.dataSkippingNumIndexedCols' = '48');

Reorder the schema. Move the columns you filter on to the front. Correct, but it usually means rewriting the table, so it is rarely the pragmatic choice on an existing large table.

The step almost everyone misses

Changing either property does not recompute anything. It changes the behaviour of future writes only. Your existing files keep whatever statistics they were written with, and until they are rewritten your queries will not improve. People make the change, see no difference, and conclude the setting does not work.

On DBR 14.3 LTS and above you can force the recomputation:

ANALYZE TABLE my_table COMPUTE DELTA STATISTICS;

Two related traps

Long string columns are truncated during statistics collection. A column holding URLs, JSON blobs or free text will have min and max values that are effectively useless for skipping, while still costing you to collect. These are good candidates to exclude from the statistics column list rather than to include.

Nested fields count toward the limit individually. For statistics purposes each scalar field inside a struct is treated as its own column, so a struct with fifteen fields consumes fifteen of your slots. A table with twelve top-level columns can already be past the boundary if several of them are nested. Map and array columns cannot have statistics collected at all.

The takeaway

Clustering and statistics are two halves of the same mechanism, and only one of them is visible in the command you run. Layout decides whether a file can be ruled out. Statistics decide whether the engine is able to ask the question in the first place. If the second half is missing, the first half is doing nothing, and nothing in the platform will raise its hand to tell you.

Before choosing clustering keys on a wide table, check that statistics exist for those columns. It takes one command and it occasionally explains months of confusing results.

Has anyone else lost time to this one? And for teams on managed tables with predictive optimization, has letting it choose statistics columns from query history worked better than the columns you would have picked yourself?

Solution Accelerator Series | Incident Investigation Using Graphistry

Tushar_Parekar — Tue, 29 Sep 2026 14:53:43 GMT

Cybersecurity investigations often require sifting through large volumes of log and telemetry data to uncover the patterns and relationships behind threat activity. The Incident Investigation Using Graphistry Solution Accelerator shows how to query investigation data, visualize connections and anomalies with graph analytics, and perform investigative analysis using natural language.

With this Accelerator, you get

Ready-to-use resources: pre-built code, sample data and step-by-step instructions ready to go in a Databricks notebook
Query investigative patterns: use SQL, Python or Scala in Databricks notebooks, with support from the Databricks Assistant to write and debug queries.
Visualize complex connections: use Graphistry on the Lakehouse to explore intricate relationships and anomalies.
Investigate using natural language: perform conversational analysis with LakehouseIQ and L.O.U.I.E ..
Work with different LLM models: use the AI Gateway to switch between third-party and self-hosted models.

🔗 Launch Solution Accelerator 👈

Architecting a Medallion Lakehouse

Khasim_1 — Sun, 27 Sep 2026 17:28:50 GMT

Hi Everyone,

Recently, I led an end-to-end Lakehouse project for a client that was struggling with manual, error-prone reporting and disconnected data silos. Their core business problem was simple yet critical: the business could not trust its own numbers. They needed a modern data platform, but we faced a strict constraint: We could not place any analytical load on their production transactional systems.

In this article, I’ll walk through the three core architectural decisions that allowed us to transform their data landscape into a robust, automated Medallion architecture.

The Ingestion Dilemma: Prioritizing Source Stability The client required up-to-date reporting, but querying their production SQL Server databases directly would have impacted their live operations.

The Decision: We implemented Change Data Capture (CDC) to stream data into our landing zone.
The Result: By reading transaction logs rather than querying live tables, we achieved near real-time data ingestion with zero performance impact on the source systems. We combined this with Auto Loader for file-based sources, ensuring our ingestion was incremental, schema-aware, and highly resilient.

The "Medallion" Strategy: Building for Evolution It’s often tempting to build shortcuts from raw data directly to final reports. However, in a real-world production environment, requirements always change.

Bronze: Our "Source of Truth" copy. Durable, replayable, and immutable.
Silver: Where the "real" work happens. We implemented logic to convert raw CDC events into a clean, historical record (SCD Type 2).
Gold: We deliberately partitioned our Gold layer into separate, business-driven data marts. By decoupling these tables, we ensure that a bug in one department’s logic—such as fulfillment tracking—doesn't impact the accuracy of another department's dashboard, like Finance.

The Truth About Referential Integrity in Lakehouses A major point of architectural debate: How do we handle foreign keys? In a high-volume streaming environment, checking referential integrity on every write doesn't scale.

The Compromise: We declared our foreign key constraints in Unity Catalog to provide metadata for the query optimizer, but we did not enforce them at write-time. Instead, we shifted that check to Data Quality as Code, alerting our team only if orphaned rows appear. This keeps the pipeline performant while maintaining full visibility into data health.

Conclusion The lesson here is that a successful Lakehouse isn't just about the tools—it’s about making the right trade-offs. By prioritizing pipeline reliability through decoupled data marts, managed CDC, and monitored constraints, we didn't just build a pipeline; we built a system that the business can finally trust.

Declaring a Primary Key RELY turned my dbt unique Test Green on Duplicate Data

ivanvyd — Fri, 25 Sep 2026 18:30:18 GMT

An open dbt-databricks feature request, #1271, asks for native RELY support on primary keys and suggests that rely: true "implicitly create the not_null and unique constraints" as "a safeguard against incorrect query results". Databricks does not enforce UNIQUE constraints, so only a uniqueness test would check the data. I tested whether dbt's unique test still works once the key says RELY.

It did not. I built two dbt models from the same source, which contains one duplicate order. Their contracts differ only in the primary key option. dbt's unique test failed on the NORELY model and passed on the RELY model.

Figure 1. The same unique test on two models built from the same rows.

WHAT THE DOCS SAY
"If RELY, Databricks may exploit the constraint to rewrite and optimize queries. It is the user's responsibility to ensure the constraint is satisfied. Relying on a constraint that is not satisfied may lead to incorrect query results."
CONSTRAINT clause, Databricks SQL reference

The setup

The source holds six orders, and order 3 appears twice with different amounts, as a replayed event would. Until #1271 lands, expression does the job: dbt-databricks appends it to the primary key DDL, so this contract creates PRIMARY KEY (order_id) RELY.

models:
  - name: orders_rely
    config:
      contract: {enforced: true}
    constraints:
      - type: primary_key
        columns: [order_id]
        expression: RELY          # the control model uses NORELY, the default
    columns:
      - name: order_id
        data_type: bigint
        constraints: [{type: not_null}]
        data_tests: [unique]
      # customer and amount columns omitted

Both models keep the duplicate: filtering on order_id = 3 returns two rows from each. I ran everything on a serverless SQL warehouse (Photon, DBSQL 2026.36) with dbt-core 1.12.3 and dbt-databricks 1.12.5. The docs say these rewrites need Photon; I did not test other compute.

Why the test passes

dbt's unique test groups by the column and returns the groups where count(*) > 1. If order_id is unique, each group holds one row and count(*) > 1 cannot be true. With RELY, the optimizer takes the key at its word and replaces the query with an empty result. The physical plans, trimmed:

-- PRIMARY KEY NORELY
PhotonFilter (n_records > 1)
+- PhotonGroupingAgg(keys=[order_id], functions=[count(1)])
   +- PhotonScan parquet workspace.rely_lab_dbt.orders_norely[order_id]

-- PRIMARY KEY RELY
LocalTableScan <empty>, [unique_field, n_records]

A larger table behaved the same way. On 1,000,010 rows in 16 files with 10 duplicated keys, the NORELY run found all 10 and read every row. The RELY run found none and read no rows, and COUNT(DISTINCT order_id) on the same table returned 1,000,010 instead of 1,000,000.

Figure 2. The same duplicate-key query in system.query.history. The RELY run opened no files.

Which checks go blind

I loaded the same six rows into tables with no key, a NORELY key and a RELY key. The first two matched in every check.

Check	No key or NORELY	RELY
*dbt unique (GROUP BY … HAVING COUNT() > 1)**	1 duplicate key	0
PySpark groupBy("order_id").count().filter("count > 1"), serverless job	1	0
COUNT(DISTINCT order_id)	5	6
LEFT JOIN to a customer table with a duplicate key, row count	8	6
*COUNT() OVER (PARTITION BY order_id) > 1**	2 rows	2 rows
DQX 0.16.0 is_unique	2 rows flagged	2 rows flagged

The fan-out check missed because the optimizer dropped the join: with a unique customer key, a LEFT JOIN cannot add rows.

Figure 3. One query over three tables with identical rows. The RELY row reports six distinct keys, no duplicates and no join fan-out.

What to do about it

Find your RELY keys. information_schema.table_constraints returns the same row for a RELY and a NORELY key. SHOW CREATE TABLE and DESCRIBE TABLE EXTENDED show the option.
Validate before you declare RELY. Check uniqueness on the staging table or incoming batch, before rows reach a RELY table, and declare RELY only where the writer guarantees uniqueness. Ingestion that can replay events does not.
Count with a window in dbt. A databricks__test_unique macro in your project replaces the built-in test on Databricks. With this version, the RELY model's test failed with 2 rows, one per duplicated row. Measure its runtime on your largest models before you switch.

{% macro databricks__test_unique(model, column_name) %}
select unique_field, n_records
from (
    select {{ column_name }} as unique_field,
           count(*) over (partition by {{ column_name }}) as n_records
    from {{ model }}
    where {{ column_name }} is not null
) counted
where n_records > 1
{% endmacro %}

DQX caught it. is_unique in DQX 0.16.0 counts with a window function and flagged both rows here and all 20 in the million-row table.
Check the check. If EXPLAIN shows no scan of the table, or query history shows 0 rows read, the check tested nothing.

The window-based checks held in these runs. I found no documentation promising they will keep working, so re-test after upgrades.

Conclusion

A RELY key changes the answers of the queries meant to verify it. Here, dbt's unique test, a PySpark groupBy, COUNT(DISTINCT) and a join fan-out check all missed a duplicate key that window-based checks caught. Whatever safeguard #1271 adds should validate uniqueness before the key says RELY, or count with a window.

Solution Accelerator Series | Entity Resolution for Public Sector

Tushar_Parekar — Wed, 23 Sep 2026 17:06:08 GMT

Public sector agencies need well-informed decisions to support the livelihood and security of citizens. The Entity Resolution for Public Sector Solution Accelerator shows how entity resolution can connect disparate data sources, reveal relationships and patterns, and create a more complete view of the entities represented in agency data.

With this Accelerator, you get

Ready-to-use resources: pre-built code, sample data and step-by-step instructions ready to go in a Databricks notebook
Create a complete 360-degree view: join data sets that represent the same underlying entity.
Improve data consistency: reduce inconsistencies to support more accurate and timely decision-making.
Unlock insights with machine learning: analyze disparate data sources to uncover additional value.
Improve interoperability: support data flow within departments, across departments and across government.

🔗 Launch Solution Accelerator 👈

Learn Databricks AI Agents | Build, Deploy & Evaluate on Databricks Platform

Tushar_Parekar — Fri, 25 Sep 2026 17:23:24 GMT

Databricks Training & Certifications

Learn Databricks AI Agents 🤖

Systems that reason, call tools, and take action - built on the Databricks Platform.

🤖 Agentic AI ⚡ Mosaic AI & Agent Bricks 🧠 Persistent memory

AI agents are systems that reason, plan, and take action on their own - calling tools, retrieving knowledge, and completing multi-step tasks. On Databricks you build, deploy, and evaluate them with Mosaic AI and Agent Bricks, all governed by Unity Catalog.

The catalog offers three AI Agents courses across two levels - from a free onboarding intro to hands-on Associate labs that give your agents durable memory. Follow the path below. 👇

1 Start here · Onboarding

Onboarding · 2H

Get Started with AI Agents on Databricks

Learn what agents are, how they differ from traditional AI systems, and their key components - then build, deploy, and evaluate your first agent with Mosaic AI and Agent Bricks through interactive demos and hands-on labs.

Free · Self-paced → Free · Instructor-led → Free · Mandarin (中文) →

Self-paced also available in English, 日本語, and 한국어.

2 Go deeper: Give your agents memory · Associate

Real agents need to remember - both within a conversation and across sessions. These hands-on courses use Lakebase as the persistence layer, with LangGraph checkpointers, MLflow tracing, and deployment as a Databricks App.

Associate · 4H

Building Persistent Memory for AI Agents with Lakebase

Give agents short-term and long-term memory with LangGraph's Checkpointer and Store, backed by Lakebase autoscaling. Build and test in notebooks, observe behavior through MLflow traces, and deploy a fully stateful agent as a Databricks App.

Free → Paid / Subscription · Lab →

Associate · 1H

AI Agents and Long-Term Memory on Lakebase

A focused, hands-on lab: distinguish the three memory categories, persist a LangGraph conversation with a checkpointer, resume a thread after a restart, and recall facts across threads with a semantic store.

Paid / Subscription · Lab →

Browse all AI Agents courses in the catalog →

Reason. Act. Remember. Learn to build AI agents the right way with official Databricks training.

Tech summit FY 2027

DPatil — Fri, 25 Sep 2026 14:05:15 GMT

Just wrapped Partner Tech Summit: four days across Genie & Agents, the Lakehouse Platform, Lakebase and Governance. The big lesson? AI doesn’t have an intelligence problem. It has a context problem. Well-governed, well-organised data is what turns it into something businesses can trust.

USE CONNECTION bypasses your MCP Service's Tool Selection and Policies

ivanvyd — Fri, 25 Sep 2026 01:06:55 GMT

I registered a mock MCP server in Unity Catalog, exposed only its read tool, attached a deny policy, and then called the delete tool anyway. What decided the outcome was access to the underlying connection, and it came from grants on the schema, not on the service.

WHAT THE EXPERIMENT SHOWED
Tool selection and service policies only govern calls through the MCP Service. An identity with USE CONNECTION called the blocked delete tool through the connection proxy.
On a schema-level connection, ALL PRIVILEGES on the schema or USE CONNECTION on the catalog also opened the proxy. A metastore-level connection opened only for a grant on the connection itself.
Proxied calls are audited as ucHttpConnectionProxiedRequest, with the caller but no tool name.

An MCP Service looks like the natural security boundary for an agent tool. You pick which tools it exposes, attach service policies, grant EXECUTE, and Unity Gateway logs each call. All of that applies to calls that go through the service. The external server also sits behind a Unity Catalog HTTP connection, and the connection has its own entry point.

Databricks says so plainly in the registration guide:

"Invoking an MCP Service requires no privilege on the underlying connection—EXECUTE on the MCP Service is sufficient. Don't grant USE CONNECTION to end users: it lets them call the external server directly through the connection, or register their own MCP Service on it, bypassing the tool selection, service policies, and auditing of your MCP Service. Reserve connection access for service authors and administrators."
— Register an external MCP server

I wanted to see what each part of that warning looks like in practice.

Figure 1. Two paths to one server. POST /ai-gateway/mcp-services/<name> runs the service's tool selection, policies and mcpCall audit. POST /api/2.0/unity-catalog/connections/<name>/proxy checks connection privileges and forwards the request; none of the service controls run.

The setup

A small FastMCP server with two tools: get_inventory (read) and delete_inventory_record, a mock write that returns "deleted": true and deletes nothing. Bearer-token protected, published through a temporary Cloudflare tunnel.
A schema-level HTTP connection, workspace.mcp_boundary.inventory_conn, which is the placement the docs recommend.
An MCP Service, inventory_readonly, with include_tool_selectors: ["get_*"].
A custom service policy (Beta) that returns DENY when the tool is delete_inventory_record.
Three service principals calling with OAuth machine-to-machine tokens:
- A has USE CATALOG, USE SCHEMA and EXECUTE on the service.
- B has the same, plus USE CONNECTION.
- C has USE CATALOG and ALL PRIVILEGES on the schema, and nothing granted on the service or the connection.

Each identity sent tools/list, a read and a delete down both paths.

Figure 2. The service as its consumers see it: one tool, get_inventory. The server behind it has two.

What each identity could reach

Identity	Service: tools/list	Service: delete	Proxy: tools/list	Proxy: delete
A (EXECUTE only)	get_inventory	-32003 blocked	403	403
B (+ USE CONNECTION)	get_inventory	-32003 blocked	both tools	deleted: true
C (ALL PRIVILEGES on schema)	get_inventory	-32003 blocked	both tools	deleted: true

Through the service, all three saw one tool, and the delete came back with the documented error, -32003 "Tool not allowed by MCP service configuration." On the proxy they split. A got 403 PERMISSION_DENIED. B and C listed both tools and ran the delete:

=== User B: EXECUTE + USE CONNECTION ===
MCP Service (Unity Gateway)  tools/list                    HTTP 200  tools=['get_inventory']
MCP Service (Unity Gateway)  call delete_inventory_record  HTTP 200  {"code": -32003, "message": "Tool not allowed by MCP service configuration."}
Connection proxy (direct)    tools/list                    HTTP 200  tools=['get_inventory', 'delete_inventory_record']
Connection proxy (direct)    call delete_inventory_record  HTTP 200  result={"sku":"SKU-001","deleted":true,"note":"MOCK - nothing was actually deleted"}

The policy never sees the proxy

For a second run I widened the tool selection to every tool, so the deny policy was the only guard left. Through the service, both A and B got the policy's reason back as a JSON-RPC error: -32003 "Deleting inventory records is not permitted through this service." B then sent the same delete to the proxy and got "deleted": true. The policy ran on the path it is attached to. Nothing consulted it on the other path. (The policy docs describe a block as an HTTP 200 response carrying a databricks_service_policy object; through this MCP Service it arrived as the JSON-RPC error above.)

Figure 3. The deny policy is attached to the service and applies to all account users. It does not apply to the connection.

USE CONNECTION can come from the schema

User C mattered more than I expected. Nobody granted C USE CONNECTION. C held ALL PRIVILEGES on the schema that contains the connection, and its proxy calls went through exactly as B's did. When I revoked that grant and left C with USE SCHEMA only, the proxy returned 403 and the service returned -32007 "Not authorized to invoke MCP service."

This follows from the docs' own advice. They recommend creating the connection at the schema level "so it's governed alongside the MCP Service." In this workspace the schema-level connection picked up schema grants, so ALL PRIVILEGES or USE CONNECTION on the schema reached the connection. The schema was also the only place I could grant it: GRANT … ON CONNECTION with the three-part name did not parse, and the connection's Permissions tab returned an error. Here, a review that only read the connection's own permission list would have missed both grants.

Figure 4. Grants on the schema, not on the connection. Two of these rows open the proxy path.

Where the connection lives decides who inherits it

To check that placement was the cause, I created a second connection to the same server at the metastore level, with its own MCP Service limited to get_*. I also added User D, who has USE CONNECTION on the catalog and EXECUTE on both services. Then I ran every identity against both connections.

Figure 5. The same grants, two placements. Grants on the schema and catalog reached the connection inside them and stopped at the one outside.

Identity	Grant that could open the proxy	Schema-level connection	Metastore-level connection
A	none	403	403
B	USE CONNECTION on the schema; on the metastore connection directly	both tools, deleted: true	both tools, deleted: true
C	ALL PRIVILEGES on the schema	both tools, deleted: true	403
D	USE CONNECTION on the catalog	both tools, deleted: true	403

Through either MCP Service, all four saw only get_inventory, and the delete returned -32003. On the proxy, the schema-level connection accepted grants from its schema and its catalog. The metastore-level connection accepted only the grant made on the connection itself, and refused everyone else with User does not have any privileges on Connection 'mcp_boundary_ms_conn'. The docs' reason for recommending the schema level is that the connection is then governed alongside the MCP Service. The same placement lets connection access arrive from the schema and the catalog above it.

Figure 6. At the metastore level the connection has its own permission list, and it was the only grant that opened the proxy.

The second bypass: your own MCP Service

The warning also covers registering another MCP Service on the same connection. That needs CREATE SERVICE too. B's first attempt failed with User does not have CREATE SERVICE on Schema. After I added that grant, B created shadow_b with no tool selection; it listed both tools and ran the delete. As the schema owner, and not a metastore admin, my list of MCP services in the schema still returned only inventory_readonly. C also created a service, but calls through it failed with Insufficient privileges on connection … to vend credentials. I can't explain that difference yet.

What the logs show

The mock server can't tell the callers apart. A's governed read and B's direct delete arrived with the same header set: User-Agent: Databricks-MCP-Proxy/1.0, the connection's one bearer token, and no header naming the caller or the path. Calls blocked by tool selection or the policy never reached it.

Databricks does record both paths in system.access.audit, at different levels of detail. A governed call is an mcpCall row with the service name and tool_name. B's blocked delete is an mcpCall row with status 403 and tool_name delete_inventory_record, even though the caller received HTTP 200 with a JSON-RPC error. A proxied call is a ucHttpConnectionProxiedRequest row with the caller and the connection, but no tool name. You can see that B used the connection. You can't see from that row that B ran a delete. Calls through shadow_b appear as mcpCall rows naming shadow_b. This query surfaces all three patterns. Replace the service name with your own:

SELECT event_time,
       coalesce(user_identity.email, user_identity.subject_name) AS caller,
       action_name,
       response.status_code AS status,
       request_params['connection_name'] AS connection_name,
       coalesce(request_params['fully_qualified_name'], request_params['mcp_service_id']) AS mcp_service,
       request_params['tool_name'] AS tool_name
FROM system.access.audit
WHERE event_date >= current_date() - INTERVAL 30 DAYS
  AND (
        action_name = 'ucHttpConnectionProxiedRequest'
     OR action_name = 'createMcpService'
     OR (action_name = 'mcpCall'
         AND coalesce(request_params['fully_qualified_name'], '')
             <> 'workspace.mcp_boundary.inventory_readonly')
  )
ORDER BY event_time DESC

In my workspace it returned 30 rows. They included every proxied request (B's and C's succeeded, A's were denied with 403), the createMcpService rows for shadow_b and shadow_c, and the mcpCall rows through both. Audit rows appeared roughly 7 to 15 minutes after the calls.

Figure 7. system.access.audit for A and B during the first run, with identities relabelled. Service calls name the tool. Proxied calls name only the connection.

The permission model to use

Consumers: EXECUTE on the MCP Service, plus USE CATALOG and USE SCHEMA. That was all A needed to use the read tool, and it reached nothing else.
Service authors and admins: USE CONNECTION, and CREATE SERVICE where they register services.
A schema-level connection: give consumer groups USE SCHEMA and USE CATALOG only. ALL PRIVILEGES or USE CONNECTION on that schema, or USE CONNECTION on its catalog, is connection access. Keep CREATE SERVICE for service authors.
A metastore-level connection: it did not inherit the schema or catalog grants I tested, so its own permission list is the one to review.
Reviews: for a schema-level connection, check grants on the connection, its schema and its catalog. Run the audit query above for proxied requests and for services you didn't create.

Conclusion

An MCP Service is a real control. In this experiment, tool selection hid the delete tool, the deny policy blocked it, and the audit log recorded the blocked attempt with the tool's name. None of that applied to the connection underneath. Any identity that could use the connection reached the server directly, and on a schema-level connection that included identities whose only relevant grant was on the schema or catalog above it.

So when you review who can reach an MCP server, start with the connection: where it lives, who holds USE CONNECTION on it, and which grants on the levels above it carry that privilege down. Give consumers EXECUTE on the service and nothing that adds up to USE CONNECTION.

Tags no Databricks: Governança, FinOps e MLOps

ThomazNeto — Thu, 24 Sep 2026 14:19:52 GMT

Tags no Databricks

Você acha que tags no Databricks são só para billing?

Em 2026, uma estratégia de tags fraca não é apenas ineficiente — é uma falha de arquitetura. Custos que explodem, dados sensíveis expostos e um MLOps caótico são apenas alguns dos sintomas.

Escrevi um artigo sobre o tema, mergulhando fundo em:
🔹 A Tríade da Governança: Ungoverned, Governed e as novas System Tags
🔹 FinOps de Verdade: A fórmula para calcular o TCO real (DBU + Infra) por tag
🔹 ABAC sem Erros: A "Regra de Ouro" da herança de tags que 99% dos times erram
🔹 Hands-On Completo: Códigos em SQL, Python (MLflow) e Terraform

Este não é um artigo superficial.
É um deep dive técnico para Arquitetos de Dados, Engenheiros e especialistas em FinOps/MLOps que se recusam a aceitar o básico.

👇 Leia o artigo completo.

link artigo

#Databricks #DataGovernance #FinOps #MLOps #UnityCatalog #DataArchitecture #Cloud

Spark Internals on Databricks

aayush_410 — Thu, 24 Sep 2026 11:20:08 GMT

Every Spark engineer eventually hits the same wall: the query works, but you don't know why it's slow, or why it suddenly got fast after a one-line config change. The gap between "I can write Spark SQL" and "I can reason about Spark's execution" is almost entirely a gap in understanding four subsystems — the Catalyst optimizer, Adaptive Query Execution, the shuffle mechanism, and Spark's memory model — plus, on Databricks specifically, how Photon layers on top of all of it. This post walks through each one, not as an API reference, but as a mental model you can use the next time a query profile looks wrong and you need to know where to look.

From SQL to Execution: The Catalyst Optimizer

When you submit a query — SQL, DataFrame API, or a CLUSTER BY DDL statement — none of it runs directly. It first becomes a tree of logical operations, and Catalyst's job is to turn that tree into something the cluster can actually execute efficiently. This happens in four phases.

Analysis resolves names. Every column reference, table reference, and function call in your query is unresolved at first — Catalyst doesn't know that f.customer_level refers to a real column until it looks up the schema of fact_profitability_actuals in the catalog. This is also where type coercion happens: if you compare a string column to an integer literal, analysis is where Catalyst decides how to cast one to match the other. A query that fails with AnalysisException failed here, before a single byte of data was touched.

Logical optimization is where the famous rule-based transformations live: predicate pushdown, constant folding, column pruning, null propagation. This is the phase that decides whether your WHERE fiscper >= ... filter gets pushed all the way down to the file scan (so files get skipped) or gets applied late (so everything gets read first). It's also where Catalyst can fold CONCAT(CAST(YEAR(CURRENT_DATE())...)) into a literal at plan time, which is why constant date-range expressions in a WHERE clause don't need to be manually pre-computed — Catalyst does it for you, once, before the physical plan is even generated.

Physical planning is where logical operations get mapped onto actual Spark execution strategies. A logical Join node becomes one of several physical join implementations — broadcast hash join, sort-merge join, shuffle hash join — and Catalyst picks based on cost estimates: table sizes, whether statistics are available, and configured thresholds like spark.sql.autoBroadcastJoinThreshold. This is the single most consequential decision Catalyst makes for a large query, and it's also the one most likely to go wrong when statistics are stale or missing, which is why ANALYZE TABLE and Delta's file-level stats matter more than most people realize.

Code generation is the last step: Spark doesn't interpret the physical plan row by row. It generates actual JVM bytecode (whole-stage code generation) for chains of operators, fusing them into a single tight loop rather than paying virtual-call overhead per row per operator. This is why a filter-then-project-then-aggregate chain in Spark can be competitive with hand-written Scala — the interpretation overhead is compiled away.

The practical upshot: when a query is slow, EXPLAIN FORMATTED isn't just diagnostic theater. It's the only way to see whether logical optimization pushed your filters where you think it did, whether physical planning chose the join strategy you expect, and — critically on Databricks — whether Photon accepted or rejected each operator, since Photon operators are annotated distinctly in the plan.

Real-Time SAP Accounts Receivable on Databricks Lakeflow Declarative Pipelines

Josue2603 — Wed, 23 Sep 2026 17:12:56 GMT

One way to keep your accounts receivable information current in real time on Databricks, with the change semantics written into the pipeline definition instead of reconstructed by whoever reads the table. Here is how the data product behind it is structured, what it exposes to whoever queries it, and where every tile on the dashboard comes from.

Imagine opening your receivables view at eleven in the morning and knowing it is current. Not current as of last night's close. Current as of the invoice that posted four minutes ago and the payment that cleared while you were reading the screen.

Now imagine the thing that makes it correct not being a pipeline somebody tuned, but a block of code you can read: this table's key is these three columns, order its changes by this clock, and when the change type is a delete, delete the row.

That is what an accounts receivable data product on Databricks can be, and it is less about speed than it sounds. What makes it work is that everything underneath is structured so a single row can be corrected in place at the moment the transaction that changes it happens. Get the structure right and freshness is something you schedule. Get it wrong and no amount of streaming saves you, because you will simply be wrong faster.

So this goes from the bottom up, using the Account Receivable Visibility data product as the worked example. Sources, layers, what it publishes, and where every tile on the dashboard comes from.

The whole shape on one page. Everything below is a closer look at one of these four columns.

The shape of the flow: SAP, then Kafka, then Databricks

Three hops, and it is worth stating them plainly before anything else, because everything after this depends on the shape.

SAP to Kafka. This hop starts with a definition, not with code. In the One Connect SAP Data Modeler you declare what a data product is, and the declaration is the deliverable: which SAP tables belong to it, how they join, and what every field is called on the way out.

That definition ships as a pair of spreadsheets, which sounds unglamorous and is the point, because a spreadsheet is a thing a functional consultant can read and argue with. For this product the first one declares the sixteen entities: source table, join fields on both sides, join type, its level in the hierarchy, a business alias, and any filter. VBAK joined to VBAP on VBELN, inner, level two, filtered to order type OR. The second declares 1,203 fields across those entities, and for each one the source table and field, the business name it is published under, whether it is a key, whether it is selectable, and a description. VBRK.VBELN becomes bill_doc, marked as key.

So the join topology and the field semantics are declared once, in a file, before anything streams. What the business events then publish to Kafka is that declaration made live. One topic per source table, created as needed, with schema registry entries and schema evolution handled at that boundary. The precise capture mechanism is a later section; for now what matters is that Kafka is where SAP data lands, not where it is queried.

And you rarely start from an empty modeler. The Onibex Marketplace lists 159 pre-modeled data products across eleven SAP modules, 97 built from tables and 60 from CDS views. The sixteen this product needs come from that catalogue, which is why the interesting work in a receivables project is the last mile rather than the first.

Kafka to Databricks. From those topics the data lands in bronze Delta tables in Unity Catalog, one per SAP source table, and everything above bronze is built by a declarative pipeline that reads the change data feed of the table below it. No orchestrator deciding what changed, no high water mark column somebody has to maintain, no nightly full reload standing in for incremental logic. The table's own commit history is the input.

Databricks to whoever asks. A dashboard, a warehouse, a CRM, a machine learning workload. They query the top layer and never touch SAP. The dashboard that ships with this product is a single Power BI report, and the rest of that list is whatever else in the organization can already read a table.

The reason to separate those three hops rather than build one pipeline from SAP to a report is that the middle hop is where the meaning gets added, and it is the part you want to write once and reuse. Which brings us to how the layers are organized.

The medallion architecture, and what sits in each layer

The middle hop follows the medallion architecture, bronze to silver to gold, and the useful way to read it here is as a count that shrinks: sixteen, then seven, then eight, then two.

Sixteen foundational data products, seven business data products, eight gold entities, five kinds of consumer.

Bronze holds sixteen foundational data products. Raw SAP tables grouped into the smallest units that mean something on their own. VBAK, VBAP and VBFA as the sales order. VBRK and VBRP as the billing document. ACDOCA, BSID, BSAD and BSEG as the journal entries. KNA1, KNVV and KNVP as the customer. PA0000 and PA0001 as personnel. MARA and MARM as the material. TCURR, TCURF and TCURV as currency. Plus eight small ones for organizational structure: sales organization, distribution channel, sales division, sales group, customer group, payment terms, region texts and the Nielsen indicator.

Those sixteen come from four different areas of SAP. Finance contributes three, order to cash four, human resources one, and organizational structure eight. That ratio is the first thing worth noticing: half the sources in a finance data product are not financial. They are the semantics that make a finance number readable by somebody who does not know SAP.

Silver holds seven business data products. One per business concept: journal entries, sales billing, sales order, currency, personnel, material master, customer master. Sixteen becomes seven because the eight organizational structure entities are not concepts in their own right, they are descriptions that get attached later, and the two receivables entities, open items and cleared items, collapse into a single journal entries product.

Gold holds eight entities. The consolidations, the two unit of measure normalizations, the two summaries, and the two that consumers actually query.

Those last two are the product's real surface. Everything else is scaffolding, and the discipline of the medallion pattern is that the scaffolding is visible and inspectable rather than buried inside one query nobody wants to open.

Notice who is on the top row of that diagram. Not only the finance team. A CRM, an operational data store, a generative AI or machine learning workload. The same two contracts serve all of them, which is only possible because they are contracts and not reports.

Two streams, because exposure is two questions

The pipeline runs two streams that never merge. That is deliberate, and it is the structural decision that matters most.

The finance stream answers what has been invoiced and not paid. It is anchored on ACDOCA, the Universal Journal line items, joined to BSEG on document, fiscal year and line item, and to BSID, open customer items, on company code, document, fiscal year, line item and customer.

Then the join that is worth stopping on. BSAD, cleared items, joins on ACDOCA.BELNR equal to BSAD.AUGBL, and on the line item. AUGBL is the clearing document. An invoice does not become paid because a flag flips; it becomes paid because a second document appears and points back at the first. Payment behavior is not a state on the invoice, it is a separate document joined on a field most people never look at.

The sales stream answers what has been committed and not yet invoiced. It is anchored on the sales order: VBAK for the header, VBAP for the items, VBFA for the document flow, VBPA for the partners, VBKD for the business data. Those are the same five tables I wrote about when describing what an SAP data product is, and this product consumes that data product rather than rebuilding it.

The sales stream ends with a definition of an open order written as three exclusions, and on Databricks you can see where the three flags come from rather than only the filter that uses them. Each one is derived from the subsequent document category in VBFA: category M sets the invoice flag, category P sets the deposit flag, category O sets the credit note flag. Everything already billed, prepaid, or credited then comes out. What remains is business the company has committed to and has not yet turned into a receivable.

Two streams, because credit exposure is not the receivables balance. It is what the customer owes plus what they are about to owe, and the second part lives in Sales and Distribution rather than in Finance.

What the product exposes: contract one

The finance stream ends in a table published at one row per journal entry line. The key is client, ledger, document number, line item, fiscal year, and company code.

What a consumer gets on that row, grouped by what it is for:

Identity and posting. Document number, line item, fiscal year, fiscal period, posting date, account, cost center, account type.

The billing link. Billing document, billing date, net value from the billing item, billing type. This is how an accounting line traces back to the invoice a customer actually received.

Receivable state. Due date, payment terms, cleared amount, clearing document and clearing date from the open items side, and the same four again from the cleared items side. Both sides are on the row, which is what makes a single query able to compare outstanding against collected without a second pass.

Measures. Quantity in the material's base unit, and amount in group currency.

Dimensions with their descriptions. Sales organization and its text. Distribution channel and its text. Division and its text. Plant. Customer number, name, city, country, region and the region's text. Customer group and its text. Sales group and its text. Four additional customer group fields. Nielsen indicator and its text.

Ownership. The personnel number of the sales representative.

Every code arrives with its word. That is not decoration; it is the difference between a table a finance analyst can query and a table that needs a lookup sheet.

What the product exposes: contract two

The sales stream ends in a second table, published at one row per open sales order item. The key is the sales order and the item.

The columns are narrower, because an open order has less to say than a posted document: order date, fiscal year and month, order type, sales unit, material, net value, currency, and the three status flags described above.

But the dimensional half is nearly identical to contract one. The same sales organization and text, distribution channel and text, division and text, plant, customer with name, city, country, region and text, the same customer group fields, the same sales representative.

That repetition is the design, not an accident.

Why the dashboard has one filter bar

The dashboard filters on company code, plant, sales organization, sales group, country, and region. Six filters, applied to both halves of the picture at once.

That only works because both contracts carry the same dimensional spine. Filter to a region and the invoiced-and-unpaid number and the committed-but-not-invoiced number both narrow, in the same way, resolving the same code to the same word.

If the two streams had been enriched independently, this is exactly where it would break. One would resolve region from the customer master and the other from the sales document, and a regional exposure figure would stop reconciling with the sum of its parts. The eight organizational structure sources are shared for that reason.

What the dashboard visualizes, and where each tile comes from

The dashboard ships as a single Power BI report, built directly on the two gold entities. It is a starting point, and a starting point only needs to exist once, because the first thing any finance team does with a delivered dashboard is change it.

Six filters across the top, six tiles, three visuals and the ledger. Every element resolves to a column in one of the two gold entities. Figures shown are from a demonstration environment.

What matters more than the tool is that nothing on that page needs a join. The report reads two tables and does arithmetic, which is what a data product is supposed to leave for the consumer.

Total receivables and overdue receivables both resolve to the open item amount. The difference between them is a comparison of the due date against today, which is why the due date and the payment terms travel on the same row as the amount.

Percent overdue is the ratio of those two, and it is the tile most likely to be wrong for structural reasons rather than arithmetic ones. Getting it right depends entirely on which date the product treats as the due date.

Days sales outstanding comes from the two dates the product ships rather than from a column named DSO: the posting date and the clearing date. The product publishes the dates. The measure is defined in the dashboard layer, which is the correct place for it, because two finance teams will define it differently and neither definition belongs baked into a pipeline.

Cash collected reads the cleared side: clearing date and cleared amount, from the BSAD half of the row.

Predicted cash is the one tile that needs both streams. Receivables coming due in the window from the finance stream, plus open orders expected to bill from the sales stream. A tile built on the finance stream alone would forecast only the money already invoiced, which understates the pipeline by exactly the amount of business in flight.

Aging analysis buckets the open items by how far past due they are. Worth being precise about what the product hands over here: it ships the due date and the payment term day count, not the buckets. The bucket boundaries are a business decision and they live in the reporting layer.

Top customers is the customer name and the amount, ranked. The name is on the row because the customer master is one of the sixteen sources.

The invoice ledger is the underlying grain made visible: document number, customer, name, posting date, due date. It is the same rows the tiles aggregate, which is why a number on a tile can be clicked down to the documents behind it.

How the data reaches Databricks

Onibex One Connect captures changes in SAP ECC and SAP S/4HANA through application-layer business events (RAP, BOR, BTE, PPF), pushed outbound over an HTTP RFC destination of Type G to the Smart Gateway, and streamed to Apache Kafka, Confluent Cloud, Databricks, Snowflake and other targets. Nothing is read from the database layer. Nothing is pulled.

That is the first hop, and for receivables the relevant property of it is not raw speed. It is that the update is triggered by the transaction rather than by a schedule, because a schedule does not know when anything happened. It only knows what time it is.

The second hop, Kafka into the medallion layers, is where the more common misreading happens. Bronze, silver and gold are frequently built as scheduled batch jobs, one layer waiting on the one below it, which is how a three-layer architecture turns into a three-window delay. Which brings us to the part of this that is specific to Databricks.

Where the change semantics are actually declared

Every silver and gold table in this pipeline is built the same way, in two moves.

The first move reads the change data feed of the bronze Delta table, not the table itself:

def cdf_reader(table):

return (

spark.readStream.format("delta")

.option("readChangeFeed", "true")

.option("startingVersion", "0")

.table(table)

)

What comes back is not rows, it is changes: each one carrying a _change_type of insert, update or delete, a _commit_version and a _commit_timestamp. The Delta table's own commit history is the change feed. Nothing had to be added to the source to produce it.

The second move turns that stream of changes back into a current-state table, and this is the declaration that matters:

dlt.apply_changes(

target="acdoca_cur",

source="acdoca_cdf",

keys=["belnr", "docln", "gjahr"],

sequence_by="_commit_timestamp",

apply_as_deletes="_change_type = 'delete'",

ignore_null_updates=True,

)

Four statements about the data, in four lines, on the table they apply to.

keys is the declared grain. ACDOCA is unique on document, line and fiscal year. BSID and BSAD need five columns, company code, document, fiscal year, line item and customer. VBAK needs one, VBELN. VBAP needs two. VBFA needs three. That is not configuration, it is the grain of the business object written down where the code that maintains it can be checked against it.

sequence_by is the clock that decides which version of a row wins when two changes arrive for the same key. This is the argument that quietly decides whether your table is right.

apply_as_deletes is the line that makes a delete a delete. Without it, a removed row in SAP leaves a row behind in silver forever, and nothing about that failure is visible, because the evidence of the problem is exactly the record that stopped arriving. I have written about the same failure happening on a different stack for the opposite reason, and the lesson generalizes: a pipeline that cannot express a delete will never tell you it has one.

ignore_null_updates keeps a partial update from blanking fields it did not carry.

The result is what the code calls a Type 1 snapshot: current state, no history. Those _cur tables are what the silver joins read, which means the join is always operating on one committed version of each source row rather than on a stream of changes it would have to reconcile itself.

The argument on clocks, and an honest inconsistency

Two things about that declaration deserve more than a passing mention, and the second one is a defect rather than a design.

The first is what sequence_by actually chooses. Ordering by _commit_timestamp orders changes by when the lakehouse committed them. Ordering by an event timestamp carried from the source orders them by when the business said they happened. Those are the same thing only while delivery is perfectly ordered, and the whole point of putting Kafka in the middle is that delivery is not guaranteed to be. If two updates to the same sales order are produced in one order and committed in the other, commit time confidently applies the wrong one last.

The second is that this pipeline does both. The finance tables sequence by _commit_timestamp. The sales order tables sequence by event_ts, the timestamp the business event carries out of SAP. Two clocks, in one pipeline, on data that ends up on the same dashboard. Each is defensible on its own; having both without a stated reason is not, and it is the kind of thing that only ever surfaces as a number nobody can reconcile. It is on the list to settle, and I am naming it here rather than writing around it.

There is also a naming detail worth knowing before you copy any of this. Databricks has renamed Delta Live Tables to Lakeflow declarative pipelines, and apply_changes is now create_auto_cdc_flow, with the same signature. The dlt module and the old function names still work and the code above still runs, but new pipelines should use from pyspark import pipelines as dp and the current names.

Where the freshness comes from, which is not the table

This is the honest difference from the same product on Snowflake, and it is worth being direct about rather than claiming parity.

On Snowflake, the freshness contract is a property of the table. Every dynamic table declares TARGET_LAG = DOWNSTREAM and defers to the one consuming it, and the final entity declares an actual target, one minute, on the object that answers the question. You can read the intended freshness of the whole pipeline out of one table definition in version control.

On Databricks there is no such line in the table definition. What decides how current the gold tables are is how the pipeline runs: continuous mode, where the pipeline stays up and processes changes as they arrive, or triggered mode, where each run processes what has accumulated and stops. That is a property of the pipeline, not of the table, and it is set where pipelines are configured rather than next to the code a reviewer reads.

The trade is real in both directions. Snowflake makes the freshness target reviewable and puts the incremental mechanics inside the service, where you accept what the planner decides to maintain incrementally. Databricks makes the change mechanics reviewable, which is what the entire section above is about, and leaves freshness to the run configuration.

Neither of those is better. They put the reviewable part in different places, and if you are choosing between them, that is the question to ask rather than which one is faster.

The same contracts on other engines

Worth saying plainly, because it is the reason any of the above is reusable: the two contracts are the product, and Databricks is one way to maintain them.

The same sixteen sources, the same grain and the same column names also ship as Snowflake dynamic tables on incremental refresh, as Flink SQL jobs with an upsert changelog over compacted Kafka topics, and as PostgreSQL layers maintained by stored procedures. What differs across them is where state lives and how the incremental update is expressed. A consumer querying the gold table cannot tell which one is running underneath, and that is the point of publishing a contract instead of a pipeline.

Databricks is the one worth writing up in this much detail because it is the one where the change semantics are written out in full, in code, next to the table they govern. The grain, the ordering clock and the delete rule are four lines somebody can review in a pull request rather than behavior you have to infer from a diagram.

Where the product stops

Four things are deliberately not in it, and a pre-modeled data product that pretends otherwise is worse than one that draws the line.

Thresholds. Nothing in the product decides that a customer at ninety percent of limit gets released and one at ninety-five does not.

Bucket and metric definitions. Aging boundaries and the DSO formula belong to the reporting layer, for the reason given above.

Currency conversion parameters. The gold sales layer converts to USD at exchange rate type M, matched on the document date, with returns negated. The target currency, the rate type and the date the rate is matched on are all choices written into the transformation. All three change the number on the dashboard, and all three need an owner.

History. The current-state tables are Type 1. The change feed that fed them retains only as far back as the source table's history does, so if you need to answer what this row said last quarter, that is a Type 2 design and a different conversation.

What the structure does guarantee is narrower and more useful than a claim about speed. Every tile traces to a column, every column traces to a source, the invoiced half and the committed half of exposure filter the same way, and the rule that decides what happens to a row when SAP changes it is four lines somebody chose rather than behavior nobody wrote down. That is what makes the six tiles worth looking at.

Technical reference: the deployment artifacts and integration steps are documented in oneconnect-docs. The Finance data products this one consumes are listed in the Onibex Marketplace.

Josue Ramirez works on data and AI at Onibex. More at onibex.com.

Databricks Data Quality - Agent Proposes, DQX Disposes

KeerthiItigimat — Wed, 23 Sep 2026 02:40:25 GMT

Every serious data platform grows a dead-letter table, and it always becomes a graveyard. Here is how we turned one into a self-healing queue — without letting a language model anywhere near the warehouse.

Every data platform that takes quality seriously ends up with a dead-letter table. You run pre-commit checks, and the rows that fail don’t go into Silver — they go into quarantine. It is the responsible thing to do. It is also, in practice, where data goes to die.

Nobody wants to open the quarantine table. The rows that land there are, by definition, the annoying ones: a code with a stray alphabetic prefix, a timestamp in the wrong format, a duplicate that got emitted twice, a foreign key that doesn’t quite line up because one side trims whitespace and the other doesn’t. Fixing them is fiddly, one-off work, and the backlog only grows.

We built a system that closes the loop. An LLM agent triages quarantined rows and proposes fixes. Those fixes are re-validated deterministically against the exact checks that quarantined the row in the first place. Only the rows that pass are merged back into Silver. Everything else escalates to a human review queue instead of looping forever.

The interesting part isn’t the LLM. It’s the constraints around it.

01
Two design principles

Everything in the system falls out of two rules we decided not to break.

“The agent proposes. DQX disposes.”
PRINCIPLE ONE

The agent never writes to Silver or Gold. Not once. Every agent-curated row is re-run through the same data-quality engine — we use DQX, the Databricks Labs quality framework — that quarantined it. If the “fix” doesn’t actually make the row pass the checks, it doesn’t get in. This keeps a hallucinated correction from silently corrupting the warehouse, and it makes the whole thing auditable: the gate is code, not vibes.

“Documented fixes beat guessed fixes.”
PRINCIPLE TWO

If a rule owner has already written down exactly how a violation should be remediated — “primary-key collision on customer_id: keep the row with the latest updated_at, drop the rest” — then that fix is applied deterministically and the LLM never sees the row. The agent is the fallback for rules nobody has written a playbook for yet. It is not the default path.

That second principle matters more than it sounds. The cheapest, most reliable, most auditable model call is the one you never make.

Figure 1. The loop. A quarantined row is fixed by a documented playbook strategy or, failing that, by the agent — then re-validated against the identical DQX checks before anything touches Silver.

02
The shape of the pipeline

The whole thing runs as a Databricks Workflow. Because a For Each task can only wrap a single task, not a sub-graph, the per-table pipeline is a child job that the parent fans out to — one child run per quarantine table.

Figure 2. The Workflow DAG. Parent discovers and fans out; each child run walks the same seven-stage path. 06 runs on at least one success of reingest or a false gate, so every table produces an audit row either way.

Figure 3. A recreation of the parent job’s run graph in Databricks Workflows — discover, then a For Each fan-out to one child run per quarantine table. Table names and durations are illustrative.

Each stage writes only to a control table — a review queue with one row per quarantined row per pass. Nothing touches Silver until the final merge, and that merge only ever sees rows that have already passed re-validation.

03
Triage

Pull the unprocessed rows from one quarantine table, dedupe to the latest quarantined version per row, and flatten the failure metadata. DQX records failures as an array of structs — {name, message, …} — and triage flattens those names into a plain array<string> of violated rule names, then seeds the review queue with status pending.

It also captures the business key at this moment. Reconstructing a natural key later, after the row has been patched, is guesswork; capturing it at quarantine time is not.

04
Playbook remediation — the deterministic half

This is where documented fixes get applied. Each dataset has a YAML file — not in the codebase, but in a Unity Catalog Volume, so adding or changing a rule needs no deploy:

Entries run in priority order; the first one that matches a row wins. Three things we learned to bake in:

where predicates let one rule route by value. status = 'Duplicate' goes to dedupe; status = 'Invalid' has no entry, so it falls through to the agent. Same rule, different handling, driven by the data.
reject_rows is a terminal state, not a failure. Some rows genuinely can’t be recovered. Dropping them is a decision the rule owner made on purpose, and it is recorded as such — not retried, not escalated, not seen by the agent.
Dedup is configured only here. The agent never deduplicates. Choosing which of two near-identical rows survives is a business call with a documented tie-breaker, not something to infer.

The strategy functions — dedupe_keep_latest, standardize_value, fill_default, reject_rows — are code. A new kind of fix needs a deploy. A new instance of an existing kind is just a file upload. Behavior in code, policy in config: that split is what keeps the playbook maintainable by the people who own the rules rather than the people who own the pipeline.

05
Agent curation — the fallback

Only rows with no playbook match reach the agent. And here is the decision that makes this effective:

“One model call per distinct violation signature. Not per row.”
THE COST MODEL

If 40,000 rows all failed the same check on the same column in the same way, that is one call. The model sees the rule name, the check function, the message, a sample of failing values, and up to five passing examples per flagged column. It returns a single deterministic transform per column, chosen from a fixed vocabulary:

Spark then applies that transform to every row in the signature. The model does not see 40,000 rows, does not write 40,000 fixes, and does not get to invent an expression. It picks lower_trim for a casing problem or regexp_replace to strip a known prefix, and that is the entire surface area. Token cost is independent of quarantine volume.

Transforms can be chained — strip a prefix, then zero-pad — which covers most of the real formatting damage: typos, column shifts, date formats, casing, padding, regex-strippable junk. And we guard against no-ops: if the model proposes left_pad on a string already at the target length, that is not a fix, and it gets downgraded to an escalation.

06
The cross-table probe — no model at all

Referential-integrity checks — a row in table A must match a row in table B — are common, and the LLM is bad at them. So before the agent is involved, a deterministic probe runs: on a sample, normalize the compared column (trim, lower_trim, upper_trim) on both sides of the join and check whether the mismatch clears. If a normalization resolves, say, 95% of the sample, propose that transform. If it doesn’t, escalate.

A surprising fraction of “broken foreign keys” are just one side of a join carrying trailing whitespace.

07
The confidence gate

The agent attaches a confidence score to every fix decision. Between curation and application, a gate checks the minimum confidence across the batch against a threshold. Above it: proceed to auto-apply. Below it: skip straight to audit, and those rows wait for a human.

One low-confidence signature holds back the whole batch for that table. We would rather under-automate than merge a shaky fix.

08
Apply and re-validate

For rows that clear the gate: apply the patch, strip every bookkeeping column, and re-run the row through approved_checks_for(silver_table) — the same rules table that quarantined it. Not a copy of the checks. Not a checks file that might have drifted. The same source. “Re-run the exact checks that quarantined it” is only true if both sides read from one place.

Rows that pass are staged. Rows that fail get retry_count += 1 and go back into the queue. After three failed attempts a row auto-escalates regardless of confidence. Nothing loops forever.

09
Reingest

A dynamic MERGE INTO Silver, keyed on the business key that triage captured, columns intersected with the target schema. Every reingested row is tagged with lineage: agent_curated = true, the confidence score, the run id, the timestamp. Downstream consumers who don’t trust agent-curated data can filter it out. That is their call to make, and we give them the column to make it with.

Figure 4. The review queue — one row per quarantined row per pass. Terminal states in green/red, human work in amber.

Figure 5. Illustrative outcome split for a single run. The playbook clears the bulk; the agent handles a couple hundred; a couple dozen reach a human.

10
Dry runs

Because “an LLM will now edit rows that flow into our warehouse” is a sentence that makes people nervous, there are two ways to see what would happen without committing anything.

A read-only estimate that computes the deterministic half for real — match, apply strategy, re-validate — and reports the agent half as a count only. The model is never called. One row per table into a report table, nothing else.
A full dry run that runs the whole pipeline, calls the agent, writes the queue and the staging table — but holds the final MERGE and instead reports the exact insert/update split against Silver.

Run the first before you turn the system on. Run the second in staging when you want to see precisely what will land.

11
What this bought us

The quarantine table stopped being a graveyard. On the first real run, the large majority of quarantined rows resolved deterministically — dedup and structural rejects that rule owners had already documented. A few hundred went to the agent. A couple dozen genuinely needed a human, and those are the ones a person should be looking at.

The lesson we keep relearning: the LLM is the least important component. The value is in the scaffolding — the deterministic-first routing, the fixed transform vocabulary, the re-validation gate, the retry cap, the confidence threshold, the dry runs. The agent is a fallback that turns “a human triages every weird row” into “a human triages the genuinely ambiguous ones.”

Agent proposes. DQX disposes. Silver stays clean.

Enterprise AI Has a Serious Context Problem

Brahmareddy — Tue, 22 Sep 2026 16:12:28 GMT

After coming back from the Databricks Data + AI Summit this year, one thought has stayed with me. I keep hearing Databricks co-founder and CEO Ali Ghodsi talk about something that I am also starting to see in my own Data + AI POCs.

For most enterprise AI use cases, the biggest problem may not be model intelligence anymore.

It is context.

Think about how much business knowledge exists inside a company.

Meetings where important decisions were made. Business rules that teams follow every day. Past decisions and exceptions. Workflows built over many years. And a lot of knowledge that simply lives inside employees’ heads.

AI does not automatically know any of this.

I have seen this while building POCs as well.

Today, connecting an LLM and building an AI agent is becoming easier. We can create an interesting demo surprisingly fast.

But then the real questions start.

Does the AI know what “revenue” actually means inside your company?

Does it know which customer definition is trusted?

Does it know which system is the source of truth?

Does it understand why a business process works the way it does?

That is where things become difficult.

The model may be very intelligent, but without the right business context, it can still give the wrong answer.

This is why I am becoming more interested in the context layer of enterprise AI.

Databricks is also moving in this direction with capabilities such as Genie Ontology, which is designed to bring business meaning and organizational context closer to data and AI.

Ali has also made another point that I think is important. Many organizations are still very early in actually using AI to automate work and create meaningful business value.

From what I am seeing, the next phase of enterprise AI may not be about finding an even smarter model.

It may be about helping AI understand how our businesses actually work.

AI already knows a lot about the world. Now we need to help it understand our organizations.

For data engineers, I think this creates an interesting opportunity.

We have spent years building pipelines that move data.

Now we may also need to help build the context that gives that data meaning.

What are you seeing in your Data + AI projects? Is context becoming a challenge for you too?

Row-level security when the filter column is not on your table

Ashwin_DSA — Mon, 21 Sep 2026 13:39:31 GMT

A common question when setting up row-level security in Unity Catalog is what to do when the column that decides who can see a row is not on the table you want to protect. The value lives in a related table, one join away. This comes up often in insurance, where a claims table is keyed by policy, but which agent services which policy is held in a separate mapping table.

This post walks through that situation. It shows why the first idea does not work, the pattern that does, and a way to set it up so one small control table drives the same rule across many tables. Every table and column name here is a plain example you can rename to your own.

Key takeaways

A row filter is a SQL function, so it can hold a subquery. That lets you scope a table by a value that lives in a related table.
You cannot filter a table and also read that same table inside another table's filter. Unity Catalog stops it with an error.
Keep the lookup in a small control table that is never filtered, and point every table's filter at it.
The filter reads the control table under the owner's rights, so users need no access to it. Onboarding a user is a one row change.

What is in this post

The problem
How row filters and ABAC work
Why filtering the mapping table does not work
How to scope the fact table
One control table for many tables
When users also need to read the mapping table
Checking it works
Things to keep in mind

The problem

Here is the model. There are three tables.

The claims fact table, claim_fact. One row per claim, keyed by policy_id. It does not carry an agent id.
The mapping table, policy_agent_map. One row per policy an agent services, holding policy_id and agent_id.
The directory, agent_directory. Maps the signed-in user, user_email, to their agent_id.

The goal is that an agent sees only the claims for the policies they service, and can also read the mapping table and see only their own policies. The value that decides all of this, the agent id, is two joins away from the claims table.

The claims table reaches its entitlement two joins away, through the mapping table to the directory, based on the signed-in user.

The objects in the schema. The claims fact and the mapping table, the directory and the control table, and the two filter functions.

How row filters and ABAC work

A row filter in Unity Catalog is a SQL function bound to a column. It runs for every row and returns true or false. If it returns true, the row is shown. A simple filter on a table that already has the agent id looks like this.

CREATE OR REPLACE FUNCTION scope_by_agent(agent_id STRING)
  RETURN is_account_group_member('claims_supervisors')
      OR agent_id = current_user();

ALTER TABLE agent_book SET ROW FILTER scope_by_agent ON (agent_id);

Attribute-based access control, ABAC, does the same job through a governed tag on a column plus a policy. You tag the column once, write one policy, and it covers every table that carries that tag, including tables added later. That is the better fit when the same rule applies across many tables. The row filter idea below works the same way whether you bind it to one table by hand or apply it through a tag and policy.

Why filtering the mapping table does not work

The first idea most people reach for is to put a row filter on the mapping table, then have the claims filter read that mapping table to find the agent's policies. This does not work.

Unity Catalog does not allow a table that already has an active row filter or column mask to be read inside another table's filter. The query fails with the error UNSUPPORTED_NESTED_ROW_OR_COLUMN_ACCESS_POLICY. So the mapping table cannot be both filtered for users and used as the lookup for the claims filter simultaneously.

A filtered mapping table cannot be read inside the claims table's filter. The two protections cannot stack this way.

Here is the error in the workspace. The mapping table already has a row filter, and the claims table's filter tries to read it.

The bind fails because the filter reads a table that already has its own filter. The call sequence in the error names both tables.

How to scope the fact table

A row filter is a SQL function, so it can hold a subquery. Bind the filter to the join key on the claims table, and resolve the agent inside the function by joining the mapping table to the directory on the current user. As long as the mapping table has no filter of its own, there is no nesting and it works for every agent.

-- The filter takes the claims table's join key, the policy id.
CREATE OR REPLACE FUNCTION scope_claims_by_policy(policy_id STRING)
  RETURN is_account_group_member('claims_supervisors')
      OR policy_id IN (
           SELECT m.policy_id
           FROM   policy_agent_map m
           JOIN   agent_directory d ON m.agent_id = d.agent_id
           WHERE  d.user_email = current_user()
         );

ALTER TABLE claim_fact SET ROW FILTER scope_claims_by_policy ON (policy_id);

The claims table in Catalog Explorer. The row filter is bound to scope_by_policy on the policy_id column.

The filter function as it sits in the catalog. Note that the security type is DEFINER, which is why the agent needs no access to the lookup.

There is a useful detail in how this runs. Row filters run with the object owner's rights, apart from the identity checks current_user() and is_account_group_member(), which run as the querying user. So an agent needs no access to the mapping table or the directory. The function reads them under the owner's rights, and only the identity check runs as the agent. The agent needs access only to the claims table.

One control table for many tables

In a real estate of tables, the same question comes up again and again on different tables. Rather than scatter lookups across many functions, put the mappings in one place. Create a small control table that holds the user and every key you scope on, and never apply a filter to it. Then point the filter for each table at that one control table.

-- One control table. Keep it locked down and never filter it.
CREATE TABLE agent_entitlements (
  user_email STRING,
  agent_id   STRING,
  policy_id  STRING
);
REVOKE ALL PRIVILEGES ON TABLE agent_entitlements FROM `account users`;

-- Filter for any table keyed by policy id.
CREATE OR REPLACE FUNCTION scope_by_policy(policy_id STRING)
  RETURN is_account_group_member('claims_supervisors')
      OR policy_id IN (
           SELECT policy_id FROM agent_entitlements
           WHERE user_email = current_user()
         );

-- Filter for any table keyed by agent id, the mapping table included.
CREATE OR REPLACE FUNCTION scope_by_agent(agent_id STRING)
  RETURN is_account_group_member('claims_supervisors')
      OR agent_id IN (
           SELECT agent_id FROM agent_entitlements
           WHERE user_email = current_user()
         );

Because the control table is a separate object that is never filtered, you can now apply a filter to any table safely, including the mapping table. Tag each key column with a governed tag and create one policy per key type in Catalog Explorer, and the rule covers every table that carries that key. You can also bind a function to a single table by hand with SET ROW FILTER. Onboarding an agent or moving a book of policies becomes a change to rows in the control table, not a change to code.

When users also need to read the mapping table

If agents only ever read the claims table, you are done. If they also need to read the mapping table directly, there are two clean choices.

A view. Leave the mapping table unfiltered, and create a view over it that filters to the current user. Give agents access to the view, not the base table. This is the smaller change.
The control table. With the control table in place, apply a filter to the mapping table directly with the scope_by_agent function above. The control table is separate and never filtered, so there is no nesting and no view to maintain.

-- The view option, if you would rather not filter the mapping table.
CREATE OR REPLACE VIEW policy_agent_map_user AS
  SELECT * FROM policy_agent_map
  WHERE is_account_group_member('claims_supervisors')
     OR agent_id IN (
           SELECT agent_id FROM agent_entitlements
           WHERE user_email = current_user()
         );

Taking the control table option, the mapping table then carries its own row filter, and it resolves through the same control table as the claims table.

The mapping table with its own row filter, scope_by_agent on the agent_id column, driven by the same control table.

Checking it works

These results are from a live run. The claims table holds 40 claims across 12 policies and 3 agents. Signed in as an agent who services 3 policies, a plain query with no where clause returns only their rows.

Claims grouped by policy for the signed-in agent. Three policies returned, 11 of the 40 claims in the table.

The same agent reading the mapping table sees only their own three policies, not all twelve.

The mapping table read by the same agent. Three of its twelve rows.

Query, run with no where clause	All rows in the table	Seen as one agent
claims	40	11
distinct policies	12	3
mapping rows	12	3

Test the all-access path too. A member of the claims_supervisors group, or a service account that runs the pipelines, gets every row. To check that path without waiting on group membership to take effect, add one row to the control table for your own user and rerun. The counts move at once. Remove the row and they return to the scoped numbers.

Things to keep in mind

The tables a filter reads must not have their own active filter or mask. Keep the control table and the directory unfiltered.
Keep the control table small so the join runs as a broadcast hash join. A few simple conditions read faster than many.
Match the function parameter type to the key column type, and keep ANSI mode on, so a bad cast raises an error rather than returning null and letting rows through.
A table can have one row filter in effect. A policy keyed on policy id and a policy keyed on agent id are separate and do not clash on the same table.
Use the key that matches the table's grain. Use the policy id for a policy-level table, or a household or account key for those grains, and tag whichever key you join on.

The shape of this pattern is the same whenever the value that decides access is not on the table you want to protect. Put the mapping in a control table, keep it unfiltered, and read it from a row filter bound to the join key. It scopes one table or a whole estate with the same small set of pieces.

Further reading in the Databricks documentation: Filter sensitive table data using row filters and column masks, Attribute-based access control in Unity Catalog, and Create a dynamic view.

I would like to hear from others, too. Have you had cases where the value that decides access sits in an awkward place, a different table, a hierarchy, or several hops away, and how did you handle it in your own projects? Share what worked for you in the comments.

Check my blog on Databricks Unity Catalog

kapil_dua — Mon, 21 Sep 2026 12:28:53 GMT

Struggling with a legacy Hive Metastore? I published a deep dive on modernizing your Data Governance on Databricks. Read how we executed a Unity Catalog migration for a complex, multi-tenant enterprise without disrupting daily business operations.

https://medium.com/@kapil.dua/navigating-multi-tenant-data-governance-transitioning-to-databricks-unity-catalog-7fd90b4fe48f

Genie Ontology na prática

ThomazNeto — Sun, 20 Sep 2026 18:31:53 GMT

Databricks - Genie Ontology 🧞‍♂️

Se "receita" significa uma coisa para Vendas, outra para o Financeiro e uma terceira no ERP, o seu agente de IA vai escolher uma sozinho. E vai responder com a mesma confiança nos três casos.

É esse o problema que o Genie Ontology tenta resolver, e montei as quatro peças do zero num workspace Databricks.

No artigo eu explico, com a documentação oficial na mão e print do meu ambiente em cada etapa:

✔️ o que é Genie Ontology, e como ele decide em qual fonte confiar quando duas se contradizem (a lógica de authority score é parecida com PageRank)
✔️ o que é um Domain, por que ele é uma governed tag por baixo, e o ganho de negócio de delimitar a descoberta por área
✔️ o que é uma Page, quais são os 7 campos dela, e o detalhe que mais me interessou: definição escrita por gente vence contexto inferido no desempate
✔️ o que é certificação, e por que ela influencia diretamente qual contexto prevalece
✔️ o Genie One citando a minha Page como fonte, com a conta conferida na mão

E o caminho de começar sem virar megaprojeto: um domínio, uma métrica crítica, um fluxo onde o time já perde tempo reconciliando número na mão.

Artigo no linkedin: https://www.linkedin.com/posts/thomaz-antonio-rossito-neto_databricks-genieontology-datagovernance-activity-7506666887947280386-i2XZ?utm_source=share&utm_medium=member_desktop&rcm=ACoAAApnG2sBB0_3dQhhqwrICkcUDtERVIHY0KE

#Databricks #GenieOntology #DataGovernance #UnityCatalog #EngenhariaDeDados #ArquitetoDeDados #AI #Agentic

Property & Casualty Insurance Lakehouse

AmitDECopilot — Sun, 20 Sep 2026 03:42:47 GMT

When building a Property & Casualty Insurance Lakehouse, one of the first things to understand is:

A Claim is not just a Claim Amount.

Consider a simple auto claim.

At different points in the lifecycle, the same claim might have:

Indemnity Reserve = $15,000
Medical Reserve = $8,000
Legal/Defense Reserve = $4,000

Then actual payments start:

Indemnity Paid = $10,000
Medical Paid = $3,000
Legal Expense Paid = $1,500

And later the carrier may recover:

Subrogation Recovery = $6,000
Salvage Recovery = $1,000

This is where the data modeling becomes interesting.

If our Lakehouse has only:

claim_id | claim_amount

we have already lost much of the business context.

Instead, the model needs to preserve the relationship between:

Policy
→ Policy Term
→ Coverage
→ Insured Risk
→ Claim
→ Claim Exposure
→ Claim Financials

And Claim Financials themselves need to distinguish:

Reserve
Payment
Expense
Recovery

with additional classifications such as indemnity, medical, legal, subrogation and salvage.

Why does this matter?

Because the Lakehouse should eventually be able to answer questions such as:

What is the outstanding reserve?

How much has actually been paid?

How much of the loss is indemnity vs expense?

How much has been recovered through subrogation?

What was the applicable coverage when the loss occurred?

What is the incurred loss at a given point in time?

What is the loss ratio by coverage, risk, geography or line of business?

This is why I find industry data models interesting.

The value isn't simply having hundreds of tables.

The value is preserving the business grain and relationships so that downstream analytics can reconstruct what actually happened.

In Part 2 of my exploration of Databricks Lakehouse Industry Data Models, I went deeper into what a P&C Insurance reference architecture could look like — from Party, Submission and Policy through Coverage, Risk, Premium, Claims, Claim Financials, Reinsurance and Catastrophe.

Full article:

https://dataengineeringcopilot.com/blog/databricks-pc-insurance-data-model

My next step is to take this from architecture to implementation.

I want to build a small P&C Lakehouse on Databricks:

Bronze → raw policy, claim and financial transactions

Silver → standardized policy, coverage, claim exposure, reserve, payment and recovery data

Gold → Policy 360, Claim 360 and insurance KPIs such as paid loss, outstanding reserve, incurred loss and loss ratio

That is where an industry data model becomes much more than an ER diagram.

It becomes the semantic foundation of the Lakehouse.

#Databricks #Lakehouse #DataEngineering #Insurance #PropertyAndCasualty #DataModeling #DeltaLake #Claims #DataArchitecture

DQX - Everything The Docs Don’t Say Yet

KeerthiItigimat — Sat, 19 Sep 2026 06:04:43 GMT

DQX Studio already has its own documentation for the rule editor, the approval workflow, the scheduler. This is what running it against real pipelines, real approvers, and a growing quarantine table made us add on top of it — and around it. It'll be out of date the next time we ship something, and that's fine.

DQX gives you check functions and an engine. DQX Studio — the UI and control plane we run on top of it — gives you a place to author, approve, schedule, and review those checks without writing a notebook. Studio itself is documented elsewhere; this isn’t that document.

This is the list of things that turned out to be missing once real people, not just a pipeline, depended on it. An analyst who wants to see the data before writing a filter. An approver who wants to know why a rule looks the way it does, six weeks later. A scheduler that has to decide, unattended, whether “the last hour” means anything for a given table. A quarantine table that needs an owner other than whoever happens to open it.

What follows, roughly in the order we hit them.

Figure 1. Three layers, one API contract. Studio's own docs cover the middle layer; this post is the outer one.

01
Preview the data before you write anything

Every rule starts with someone looking at the table. The preview panel in the rule editor runs a live sample against Unity Catalog under the caller’s own identity — on-behalf-of, so row filters and column masks apply the same way they would to any other query that person runs — with two things layered on:

A column picker to narrow a wide table down to what’s relevant before you start authoring a check against it.
A natural-language filter box. Type “orders from the last week with a null customer_id” and it’s handed to a Foundation Model with the table’s real column list and a hard constraint — use only these columns, always cap the row count, default to a plain SELECT * if the request doesn’t resolve to anything sensible. The model returns SQL, not an answer; if the call fails for any reason, the preview falls back to an ordinary unfiltered sample rather than erroring out.

Small feature, but it collapses the gap between “I can describe the bad rows” and “I can write a SQL predicate for them” — which is most of the distance between a domain expert and a rule.

02
Two ways to run a check, on purpose

Databricks Apps don’t run Spark. So every dry run has to leave the app process one way or another — and we deliberately built two different exits for it, because “fast” and “exactly how it’ll run in production” are different needs.

Run Locally executes against a small sample through a local, serverless Spark session — a few seconds, no job to wait on. Good for iterating on a rule while you’re still writing it.
Run as Workflow submits the same check to the actual task-runner job — the identical path a schedule will use in production, including the real sample size and Row Scope. Slower, but it’s not a simulation of the production path, it is the production path.

An author uses the first ten times while shaping a rule and the second once, right before submitting it for approval, to confirm it survives contact with the real execution environment.

03
Every rule remembers how it was made

A rule authored six weeks ago, by someone else, using a phrase like “flag orders where the region code doesn’t match our list” — an approver reviewing that later shouldn’t have to reverse-engineer intent from a YAML blob. Two things carry that context forward: a full audit trail (dq_quality_rules_history) recording every version, every status transition, and who made it, not just the current state; and, for AI-generated checks specifically, the original natural-language instruction travels with the rule as metadata rather than being discarded once the check function comes back. The prompt that produced a rule is part of the rule’s own record, not a one-time scratch input.

04
Global rules, for the check that isn’t about one table

Most checks are bound to a table — orders.customer_id is not null. Some aren’t really about a table at all; they’re about a shape: “any column named email should look like an email,” regardless of which of forty tables it shows up in. Global rules are authored once, table-independent, and apply wherever a matching column exists — without re-authoring the same check forty times or drifting slightly different each time someone copies it.

05
Cross-table checks, and what production found

DQX’s sql_query check is the right tool for anything that isn’t row-local — a duplicate key, a referential mismatch between two tables. It also has a real contract: an explicit condition column, and merge_columns to join violations back to specific input rows. An author writing “give me the rows that violate this” doesn’t know that contract, and shouldn’t have to.

what an author writes vs. what DQX's engine requires

The editor derives that wrap on save, and only trusts a column as a merge key once it’s checked against the target table’s real schema — a renamed join column looks safe in the query text and fails at execution time, so schema-text alone isn’t enough. A related fix went the other way: for a query that doesn’t actually depend on its home table’s rows (its own independent join), the engine was pre-sampling that input before the check ran, which could silently drop real violations whose key wasn’t in the sampled slice. The fix routes that shape through the same fast path a true cross-table check already used, instead of teaching the row-level engine a special case.

Underneath both: Unity Catalog metadata reaching a fresh serverless job can lag well behind a SQL warehouse seeing the same object, so the read path needed a retry budget sized off actual observed timelines, not a guess.

06
Alerting that isn’t just “check the app”

A validation result nobody sees is worth nothing. Two channels out:

Microsoft Teams webhooks, scoped per channel by trigger — all runs, scheduled-only, or manual-only — so a channel used for production monitoring doesn’t fill up with every dry run an author fires off while iterating.
Site24x7, or any uptime monitor, via a pair of read-only status endpoints — latest result by table, or by a specific run id — that return a plain 200 on a clean run and a 503 the moment error rows show up. External monitoring infrastructure gets to treat data quality the same way it treats an HTTP health check, without knowing anything about DQX.

07
Schedules that know what they’re sampling

“Validate the last hour” only means something if the scheduler knows which column is time, and on a table it’s never seen before, that’s a real guess. It tries typed timestamp/date columns first, then a name-priority list, and surfaces the candidates as editable tags — a schedule owner can pin the right one, remove one that’s wrong for their table, or leave it on automatic. Row Scope pairs with tag-based scope filtering, so one schedule can target “every approved rule labeled pii” across every table that has one, on either the full table or a time-windowed sample, instead of one schedule per table.

08
Closing the loop, not just quarantining

Everything above eventually produces failing rows, and Studio’s side of what happens to them is deliberately small: a per-dataset YAML runbook, edited or uploaded in the app, written straight to a Unity Catalog Volume. The app checks that it’s valid YAML and writes it — it doesn’t parse rule names, strategies, or parameters, so a new remediation strategy is a change to the consuming pipeline, never to Studio.

What reads those runbooks — an agentic re-ingestion system that tries the documented fix first and falls back to an LLM only for what nobody has written a playbook for — is its own build, written up separately:

Agent Proposes, DQX Disposes →

09
dqx-validator — the write side of the dead-letter pattern

Everything so far assumes rows are already in a Silver table by the time DQX sees them. dqx-validator is the piece that runs earlier — a standalone library an ingestion pipeline calls directly, mid-flight, to apply Studio’s own approved rules to a DataFrame before it lands, and quarantine whatever fails.

It loads status = 'approved' checks from the same rules tables Studio’s approval workflow writes to, resolved per-environment from a small table config. The loader is exposed as a single function, approved_checks_for(table_fqns), and the agentic re-ingestion side calls the exact same function — so the checks that quarantined a row and the checks that later re-validate a proposed fix are guaranteed to be the same object, not two copies that can drift.

Splitting the DataFrame is one call to DQX’s own apply_checks_by_metadata_and_split — no custom validation logic, just the public engine. The quarantine row carries the input schema unchanged plus bookkeeping columns (_error, _warning, _generated_at, _data_source, _rule_sources, _is_cross_table), and one more that matters more than it looks: _row_id, a hash of whichever business-key columns the caller names. That’s the field the entire review queue on the read side keys every quarantined row on. Name the wrong columns, or none, and the hash falls back to every input column — which still works, but logs a warning, because at that point two rows differing only in a bookkeeping timestamp look like two different violations instead of one.

Figure 2. Items 08 and 09 are two ends of the same loop — dqx-validator writes to quarantine, the remediation pipeline reads it back out, and both sides resolve rules the same way.

10
Ask it directly, instead of building another chart

Studio has an Insights dashboard for the questions we anticipated. It doesn’t have one for the question someone asks once, in passing — “which tables had a spike in errors last Tuesday” isn’t worth a dashboard tile, but it’s a completely reasonable thing to want to know. A Genie space scoped to Studio’s own schema — run history, quarantine records, metrics — turns that into a conversation instead of a feature request: ask in plain language, get an answer grounded in the real tables, no ticket filed to add a chart nobody will look at again.

11
What all of this has in common

None of the ten items above touch the DQX library. Every one of them is a caller of its public API, the same way a notebook is — the editor, the validator library, the agentic pipeline, all of it sits beside DQX rather than inside a fork of it. That was never an accident; it’s the only way any of this stays upgradeable as the library keeps moving.

Studio is documented. This is everything the documentation doesn’t say yet.

Unmasking the Connected Consumer

KeerthiItigimat — Sat, 19 Sep 2026 05:47:08 GMT

Behind every clean “Golden Record” is a messy web of shared emails, phones, and legacy IDs. Here’s how we turned that web into an interactive, cluster-aware graph a data steward can actually work with — built on Streamlit, NetworkX, and a Databricks SQL Warehouse.

In the world of Master Data Management, the “Single View of the Customer” is the holy grail. But behind every clean Golden Record lies a messy web of raw data: five different email addresses, shared phone numbers, and legacy system IDs that may or may not belong to the same person.

Identifying these links is identity resolution. Visualizing them, however, is where the real insights happen — and where we built the Entity Linkage Explorer: a Streamlit app that lets data stewards traverse identity clusters, isolate specific data “islands,” and filter out noisy attributes in real time, all against a live Databricks SQL Warehouse.

Figure 1. The Entity Linkage Explorer running as a Databricks App — the sidebar drives what the graph shows; the canvas renders only the clusters the steward has toggled on.

01
From tables to topology

Traditional data quality tools give you a table. But tables are terrible at showing transitive relationships. If Person A shares an email with Person B, and Person B shares a phone number with Person C, a table makes it hard to see that all three might be the same household.

Tables show rows. They don’t show the shape of a relationship.

The core problem

We needed a tool that could:

Query scale. Pull deep lineage from a Databricks SQL Warehouse without dragging the whole table across the wire.
Identify clusters. Automatically group connected individuals into “islands.”
Handle noise. Dynamically exclude junk data — test@test.com, 123 Main St — that causes false links.
Perform. Render hundreds of nodes without crashing the browser.

02
A four-tier, graph-first architecture

The app separates security, data, logic, and rendering into their own tiers — each one only doing the job it’s good at.

Figure 2. The app captures the user’s OBO token to establish a secure session, pulls only the columns needed for linking, builds a logical graph with NetworkX, then hands it to PyVis to render.

To keep the data tier fast, we avoid SELECT * and pull only the attributes necessary for linking. Before drawing anything, we build a logical graph in NetworkX: every person and every attribute — an email, a phone number, an address — becomes a node, and a shared attribute becomes an edge between the people who share it. Running Connected Components over that graph instantly identifies every isolated cluster in the dataset, no matter how many hops apart the shared attribute is.

03
Cluster filtering tames the hairball

One of the biggest pain points in graph visualization is the “hairball effect” — too many nodes create an unreadable mess. So the app never renders anything until the user has chosen what to look at.

Connected components are computed once, then listed in the sidebar as named, countable clusters:

Users toggle clusters on or off, and only the visible nodes are ever handed to PyVis. If a specific email address is stitching together two families that shouldn’t be linked, the steward can see exactly which cluster it sits in and reach for the Unlink / Exclude feature to remove it — without touching anyone else’s view.

Figure 3. Excluding one bad phone number splits a falsely merged household into two correct clusters — the exact “What-If” check a data steward runs before trusting a linkage.

04
Engineering for performance

Thousands of identity links can turn naive Python loops into the bottleneck. Three specific optimizations keep the UI snappy.

itertuples() instead of iterrows()

iterrows() is notoriously slow because Pandas builds a fresh Series for every row. Switching to itertuples() improved graph-building speed by nearly 50x, processing thousands of links in milliseconds.

Two-layer caching

Streamlit’s @ST.cache_data is split into two layers: a data cache for the raw SQL results, and a graph cache for the NetworkX object itself. Changing a purely visual setting — background color, node size — redraws the existing graph instead of re-querying Databricks or reprocessing the DataFrame.

Physics stabilization

Graph simulations are CPU-intensive. Configuring PyVis’s forceAtlas2Based solver with a high stabilization threshold lets the layout settle before render, instead of animating a bouncing simulation that freezes the browser on large loads.

pyvis network options

net.set_options("""
{
  "physics": {
    "solver": "forceAtlas2Based",
    "stabilization": {"iterations": 150}
  }
}
""")

05
Handling data quality on the fly

Data is rarely perfect. A generic placeholder like NA or a shared “house” phone number will link thousands of unrelated people together if you let it. The Explorer includes a Dynamic Exclusion engine: type any value into a “Values to Remove” box, and the app instantly:

Drops those nodes from the NetworkX graph.
Recalculates the clusters.
Redraws the now-cleaned visualization.

This is the point of the whole tool: entity linkage isn’t just an algorithm, it’s human-in-the-loop verification. NetworkX supplies the analytical power; Streamlit supplies the rapid UI. Together they turn a static lineage report into something relationships can be explored, cleaned, and understood at a glance.

Better data quality. Fewer duplicate customers. A much clearer picture of who
your consumers actually are.

Community Articles topics

Row-level security for a RAG agent: what Unity Catalog enforces, and what you have to build

The 32nd Column: Why Your Z-Order or Clustering Key May Be Doing Nothing

Solution Accelerator Series | Incident Investigation Using Graphistry

With this Accelerator, you get

Architecting a Medallion Lakehouse

Declaring a Primary Key RELY turned my dbt unique Test Green on Duplicate Data

The setup

Why the test passes

Which checks go blind

What to do about it

Conclusion

Solution Accelerator Series | Entity Resolution for Public Sector

With this Accelerator, you get

Learn Databricks AI Agents | Build, Deploy & Evaluate on Databricks Platform

Tech summit FY 2027

USE CONNECTION bypasses your MCP Service's Tool Selection and Policies

The setup

What each identity could reach

The policy never sees the proxy

USE CONNECTION can come from the schema

Where the connection lives decides who inherits it

The second bypass: your own MCP Service

What the logs show

The permission model to use

Conclusion

Tags no Databricks: Governança, FinOps e MLOps

Spark Internals on Databricks

Real-Time SAP Accounts Receivable on Databricks Lakeflow Declarative Pipelines

The shape of the flow: SAP, then Kafka, then Databricks

The medallion architecture, and what sits in each layer

Two streams, because exposure is two questions

What the product exposes: contract one

What the product exposes: contract two

Why the dashboard has one filter bar

What the dashboard visualizes, and where each tile comes from

How the data reaches Databricks

Where the change semantics are actually declared

The argument on clocks, and an honest inconsistency

Where the freshness comes from, which is not the table

The same contracts on other engines

Where the product stops

Databricks Data Quality - Agent Proposes, DQX Disposes

01Two design principles

02The shape of the pipeline

03Triage

04Playbook remediation — the deterministic half

05Agent curation — the fallback

06The cross-table probe — no model at all

07The confidence gate

08Apply and re-validate

09Reingest

10Dry runs

11What this bought us

Enterprise AI Has a Serious Context Problem

Row-level security when the filter column is not on your table

The problem

How row filters and ABAC work

Why filtering the mapping table does not work

How to scope the fact table

One control table for many tables

When users also need to read the mapping table

Checking it works

Things to keep in mind

Check my blog on Databricks Unity Catalog

Genie Ontology na prática

Property & Casualty Insurance Lakehouse

DQX - Everything The Docs Don’t Say Yet

01Preview the data before you write anything

02Two ways to run a check, on purpose

03Every rule remembers how it was made

04Global rules, for the check that isn’t about one table

05Cross-table checks, and what production found

06Alerting that isn’t just “check the app”

07Schedules that know what they’re sampling

08Closing the loop, not just quarantining

09dqx-validator — the write side of the dead-letter pattern

10Ask it directly, instead of building another chart

11What all of this has in common

Unmasking the Connected Consumer

01
Two design principles

02
The shape of the pipeline

03
Triage

04
Playbook remediation — the deterministic half

05
Agent curation — the fallback

06
The cross-table probe — no model at all

07
The confidence gate

08
Apply and re-validate

09
Reingest

10
Dry runs

11
What this bought us

01
Preview the data before you write anything

02
Two ways to run a check, on purpose

03
Every rule remembers how it was made

04
Global rules, for the check that isn’t about one table

05
Cross-table checks, and what production found

06
Alerting that isn’t just “check the app”

07
Schedules that know what they’re sampling

08
Closing the loop, not just quarantining

09
dqx-validator — the write side of the dead-letter pattern

10
Ask it directly, instead of building another chart

11
What all of this has in common

01
From tables to topology

02
A four-tier, graph-first architecture

03
Cluster filtering tames the hairball

04
Engineering for performance

05
Handling data quality on the fly