Nobody told the analytics team they couldn't query the raw customer table. So, they did.
Full names, email addresses, phone numbers - exported to a CSV for "a quick look." No alert fired. No one flagged it. We found out three weeks later during a compliance review.
The pipeline was solid. Months of work. Clean transformations, reliable runs, well-structured tables. We just never thought about who could actually reach in and pull the data out.
That's the part that gets skipped. You build a great pipeline and assume the work is done. But in retail - where you're sitting on customer PII, order history, and payment data - the pipeline is only half the job. The other half is knowing exactly who can see what, where data came from, and having an answer ready when compliance asks.
"We trust our team" doesn't hold up in an audit.
Unity Catalog is what fixed this for us. I've been running it in production for about a year, and the biggest change isn't a feature - it's that access control stops being something you bolt on after the pipelines are built and becomes part of how the platform works. This post covers the three things I use most: PII tagging, lineage tracking, and row-level security. All with working SQL you can adapt directly.
A quick picture of what Unity Catalog does
Unity Catalog introduces a three-level namespace on top of your existing Databricks workspace:
catalog
└── schema (database)
└── table / view / volume
So instead of database.table, you now reference catalog.schema.table - something like retail_prod.sales.orders. This might feel like extra typing at first, but it's what makes centralized governance possible - a single catalog with one permission model covering all your workspaces.
Tagging PII columns - know what you're carrying
The first thing I did when setting up Unity Catalog was tag every column that carries personally identifiable information. Not because anyone asked me to - just because it's harder to lock down data you haven't mapped.
UC has a built-in tag system - catalog, schema, table, or column level. For PII I go straight to column level - it's the most precise, and it gives you a queryable inventory of sensitive fields across your entire platform.
Here's how to create and assign a PII tag:
-- sql
-- Create a tag in your catalog
CREATE TAG IF NOT EXISTS pii_category ALLOWED_VALUES 'name', 'email', 'phone', 'address', 'payment';
-- Apply tags to sensitive columns
ALTER TABLE retail_prod.customers.profiles ALTER COLUMN full_name SET TAGS ('pii_category' = 'name');
ALTER TABLE retail_prod.customers.profiles ALTER COLUMN email_address SET TAGS ('pii_category' = 'email');
ALTER TABLE retail_prod.customers.profiles ALTER COLUMN phone_number SET TAGS ('pii_category' = 'phone');
ALTER TABLE retail_prod.orders.transactions
ALTER COLUMN card_last_four SET TAGS ('pii_category' = 'payment');
Once tagged, this query gives you a full map of every sensitive column across your platform:
-- sql
SELECT
table_catalog,
table_schema,
table_name,
column_name,
tag_value AS pii_type
FROM system.information_schema.column_tags
WHERE tag_name = 'pii_category'
ORDER BY table_catalog, table_schema, table_name;
Run that and you have a full inventory of sensitive columns. The first time compliance asked us where customer email appeared across the platform, that query answered it in about ten seconds.
Worth being clear on though: tags are metadata only - they don't restrict access by themselves. What they do is give you the foundation to build policies on top of, which is where the next two sections come in.
Lineage - where did this data come from?
Every time a Delta Live Tables pipeline runs - or any notebook, job, or SQL query that reads from and writes to UC-managed tables - Unity Catalog automatically captures the data flow. No configuration required. You get column-level lineage out of the box.
The real pain shows up when an analyst reports the lifetime_value column looks off. Without lineage you're manually tracing through notebooks and pipeline code trying to figure out what fed it. With lineage, you open Catalog Explorer, click the column, and see the exact chain - Silver table, Bronze table, raw source file. Done in seconds instead of an hour.
The same lineage is queryable directly if you need it in a script or dashboard:
-- sql
-- Find all upstream tables feeding into the Gold customer table
SELECT
source_table_full_name,
target_table_full_name,
created_at
FROM system.access.table_lineage
WHERE target_table_full_name = 'retail_prod.gold.customer_lifetime_value'
ORDER BY created_at DESC;
Lineage also works in reverse. If you're about to deprecate a Bronze table, you can check what downstream assets depend on it before you touch anything:
-- sql
-- What tables depend on our Bronze orders table?
SELECT
source_table_full_name,
target_table_full_name
FROM system.access.table_lineage
WHERE source_table_full_name = 'retail_prod.bronze.raw_orders'
ORDER BY target_table_full_name;
No more "I deleted a table and three pipelines broke" incidents.
Fine-grained access control - who sees what
Permissions in UC are just SQL. You grant privileges at any level of the hierarchy - catalog, schema, or table - and they inherit downward:
-- sql
-- Analytics team gets read access to Gold only
GRANT USE CATALOG ON CATALOG retail_prod TO `analytics-team`;
GRANT USE SCHEMA ON SCHEMA retail_prod.gold TO `analytics-team`;
GRANT SELECT ON SCHEMA retail_prod.gold TO `analytics-team`;
-- Data engineers get full access to Bronze and Silver
GRANT ALL PRIVILEGES ON SCHEMA retail_prod.bronze TO `data-engineering`;
GRANT ALL PRIVILEGES ON SCHEMA retail_prod.silver TO `data-engineering`;
-- Block direct access to raw PII table
REVOKE SELECT ON TABLE retail_prod.customers.profiles FROM `analytics-team`;
The analytics team can query all Gold tables. They cannot touch raw Bronze data or the customers PII table directly. Clean separation, enforced at the platform level.
Row-level security with dynamic views
Table-level permissions are a blunt instrument. Sometimes you need more precision - regional managers should only see orders from their own region, or a customer support team should only see records for accounts they're assigned to.
The trick is is_account_group_member() - it checks the caller's group at query time, so the same view returns different rows for different people:
-- sql
CREATE OR REPLACE VIEW retail_prod.sales.regional_orders AS
SELECT
order_id,
order_date,
customer_id,
product_sku,
order_amount,
region
FROM retail_prod.silver.orders
WHERE
CASE
WHEN is_account_group_member('region-apac') THEN region = 'APAC'
WHEN is_account_group_member('region-emea') THEN region = 'EMEA'
WHEN is_account_group_member('region-us') THEN region = 'US'
WHEN is_account_group_member('data-engineering') THEN TRUE
ELSE FALSE
END;
-- Grant access to the view, not the underlying table
GRANT SELECT ON VIEW retail_prod.sales.regional_orders TO `analytics-team`;
Same query, same view - different results based on who's asking. The underlying Silver table stays locked down.
Column masking for PII
One more pattern I use heavily: column masking. Instead of hiding an entire column, you partially mask it depending on who's querying. We use this for customer support - they need to verify who someone is, but there's no reason they should be able to export a raw email list.
-- sql
-- Create a masking policy
CREATE MASKING POLICY retail_prod.security.email_mask AS (email STRING)
RETURNS STRING ->
CASE
WHEN is_account_group_member('data-engineering') THEN email WHEN is_account_group_member('customer-support')
THEN CONCAT(LEFT(email, 2), '****', SUBSTRING_INDEX(email, '@', -1))
ELSE '****@****.***'
END;
-- Apply the masking policy to the column
ALTER TABLE retail_prod.customers.profiles ALTER COLUMN email_address SET MASKING POLICY retail_prod.security.email_mask;
A data engineer sees the full email. A customer support agent sees jo****@gmail.com. Anyone else sees ****@****.***. One table, one policy, three different experiences - and zero copies of the data floating around.
Auditing - who actually accessed what
There's one more thing most people skip: checking whether any of this is actually being used. Unity Catalog logs every access event to the system audit tables:
-- sql
SELECT
user_identity.email AS user_email,
action_name,
request_params.table_full_name AS table_accessed,
event_time
FROM system.access.audit
WHERE request_params.table_full_name = 'retail_prod.customers.profiles'
AND event_time >= CURRENT_TIMESTAMP - INTERVAL 7 DAYS AND action_name IN ('SELECT', 'READ')
ORDER BY event_time DESC;
Run this weekly, pipe it into a Databricks SQL dashboard, and you have an access audit trail that satisfies most compliance requirements without any custom logging infrastructure.
The mistakes I'd save you from
Don't skip the catalog hierarchy design. Once you have tables in Unity Catalog, restructuring the catalog/schema layout is painful. Spend an hour upfront deciding how you'll organize catalogs - by environment, by domain - before you start creating tables.
Tags alone don't protect data. I've seen teams tag all their PII columns and then consider the job done. Tags are discovery and documentation - they don't enforce anything. Pair them with masking policies and view-based access control.
is_account_group_member checks group membership at query time. This is a feature, not a bug - add a user to a group and their access updates immediately without changing any views or policies. But it also means if you remove someone from a group, they lose access instantly. Make sure your group membership is managed carefully. I'd rather have a slightly annoying offboarding checklist than discover an ex-employee still has access to the customer table three months later.
Test your dynamic views as a non-admin. It's easy to build a row-level security view, test it as an admin - who often bypasses restrictions - and ship it thinking it works. Always verify by impersonating a user in the target group.
What this actually changes
The part that surprised me most wasn't the technical setup - it was how much easier governance conversations became once everything lived in the platform rather than in a spreadsheet. People can't work around it accidentally. Access is granted explicitly, audited automatically, and revoked cleanly.
In retail, where you're handling customer PII and payment data, getting this wrong isn't just a technical problem. It shows up in audits, in compliance reviews, in the conversation nobody wants to have at 9am on a Monday after a data breach.
If I had to do this again from scratch, I'd do it in this order:
- Tag PII columns first - you need to know what you're protecting before you can protect it
- Get lineage working early - it's much more useful for preventing incidents than for investigating them after the fact
- Build access control into views and masking policies, not just table-level grants
- Check the audit logs regularly, even when nothing seems wrong - that's usually when you find something useful
If you're setting up Unity Catalog for the first time or migrating from the legacy Hive metastore, start small. Pick one domain, one catalog, get the permissions right, then expand. Trying to govern everything at once leads to over-complicated structures that nobody maintains.
Drop a comment if you've hit any Unity Catalog edge cases in production - particularly around external tables or cross-workspace sharing. Always curious what others have run into.
