topic Re: ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)? in Data Engineering

ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

excavator-matt — Thu, 27 Nov 2025 18:44:56 GMT

Hi!

We're using Spark Lakeflow Declarative Pipelines for ingesting data from various data sources. However, in order to achieve compliance with GDPR, we are planning to start using ABAC tagging.

However, I don't understand how we are supposed to use this implement this on streaming tables with version control. The documentation is doesn't mention it.

I think you can tag the tables manually, but that might risk getting lost as tables are recreated. I also consider simply not allowing free access in bronze and apply tagging in version controlled models, but that seems harsh.

Have I missed something or is the support lacking here?

Re: ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

ManojkMohan — Thu, 27 Nov 2025 20:49:39 GMT

@excavator-matt

“Can we tag streaming tables with ABAC and expect it to be safe across versions?”

Yes, streaming tables are fully subject to UC ABAC, but if the table is physically recreated, table‑level tags can be lost

“Is there first‑class support in Lakeflow for this?”

Right now, the docs do not show any integration layer that manages governed tags https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-streaming-table

“What is a sane pattern?”

Use Unity Catalog everywhere for Lakeflow
Put GDPR‑relevant tags and ABAC policies at catalog leve
, manage table/column tags via IaC or deployment jobs that re‑apply tags after pipeline changes,

ABAC policies work on Unity Catalog tables
But Lakeflow Declarative Pipelines do not version governed tags for you, so you must manage tags and policies at the UC layer

Solution thinking:

For GDPR‑style constraints attach governed tags at catalog level
ABAC policies can key off both governed tags and other attributes like table name
For streaming models that are versioned (for example, customer_latest_v1, customer_latest_v2), you can keep them in a “sensitive” schema (tagged as PII) and have policies that apply uniformly,
Automate tags as part of CI/CD for Lakeflow

If you share a bit about how you currently version streaming tables (e.g., drop/recreate vs ALTER vs new names), a more pipeline‑specific tagging workflow can be solutioned

Re: ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

excavator-matt — Sat, 29 Nov 2025 06:52:45 GMT

One way to get version control might be to use the Terraform resource entity_tag_assignment. I am not sure if it supports governed_tags, but I'll experiment in the coming weeks.

This separates the version control on where the tags are defined and where the Declarative Pipeline code, but at least it is version control and you don't have to write and maintain something yourself.

Re: ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

excavator-matt — Mon, 08 Dec 2025 07:40:09 GMT

Correction. Trying this will result in this error
> ABAC policies are not supported on tables defined within a pipeline. Remove the policies or contact Databricks support.

So it isn't supported 😞