lingareddy_Alva
Esteemed Contributor

Hi @David_Dabbs ,

This is a well-structured problem. Let me address each of the three concerns systematically, then recommend an overall pattern.

Notification: Delta Live Tables Triggers vs. Recommended Alternatives

Table triggers on VIEWs are not the right tool here. Databricks does not support DML triggers (in the Oracle sense) on Delta tables or views. What Databricks does have is:

- Structured Streaming with trigger(availableNow=True) — a consumer-side poll that runs a job on a schedule and processes only new data since the last checkpoint. This is still polling, just abstracted.
- Delta Live Tables — pipeline-oriented, not event-driven across workspace boundaries.
- Databricks Workflows "File Arrival" / table-based triggers — these watch for new files in cloud storage, not Delta table writes.

The recommended pattern for cross-workspace, near-real-time notification is Databricks Workflows plus a lightweight event bus. Given your Unity Catalog shared governance, there are two practical approaches:

Option A: Workflows REST API (Job Triggering)
The producer workspace, upon completing its data load, calls the Databricks REST API to trigger a job in each consumer workspace. Each consumer pre-creates a Workflow job (notebook task) and grants the producer's service principal `CAN_TRIGGER` permission on it. The producer calls `POST /api/2.1/jobs/run-now` against each consumer workspace endpoint.

- Near-immediate, no polling
- Native to Databricks, no external infrastructure
- Manageable with your low consumer count
- Producer must know each consumer's workspace URL and job ID (a small config table suffices)
- Bi-lateral trust: consumer workspaces must authorize the producer SP

Option B: External Event Bus (AWS EventBridge or SNS/SQS)
The producer publishes an event to AWS EventBridge or an SNS topic upon data load completion. Each consumer workspace has a Databricks Workflow with an HTTP webhook trigger or a Lambda-to-REST API bridge that fires the consumer job.

- Fully decoupled
- Auditable event history
- Scales to more consumers without producer config changes
- Requires AWS infrastructure outside Databricks
- More moving parts for a low-consumer scenario

For your stated scale (low consumer count, same UC governance domain), Option A is the pragmatic recommendation. The producer maintains a small config table of {consumer_workspace_url, job_id, sp_token_secret_scope} and fans out job triggers upon completion.

Delta Share Facade: Views Are the Right Call
Your plan to expose a VIEW over the control table and the timeseries table via Delta Sharing is correct and idiomatic. A few reinforcing points:
Grant access to the consumer automation service principals, not user accounts, so permissions survive personnel changes. The view facade lets you control exactly which columns and rows consumers see — useful for isolating `is_complete` or `data_as_of_date` without exposing internal control columns. On the consumer side, the shared tables are read-only by definition in Delta Sharing, which cleanly enforces the separation of concerns.

Posting Results Back: The Recommended Pattern
Databricks has no analog to Oracle's table-valued procedure parameters callable cross-workspace. The options from most to least recommended:

Recommended: Consumer-Owned Shared Table, Producer Pulls
Each consumer workspace has a results Delta table (e.g., `consumer_ws.reporting.results_outbox`) shared back to the producer via a reverse Delta Share. The producer has a lightweight Workflow job — triggered by a callback the consumer makes when done — that reads from the shared table, validates, and writes to its own `producer.governance.consumer_results` table.
The callback is simply: when the consumer job finishes, it calls `POST /api/2.1/jobs/run-now` on the producer's ingest job, passing the `consumer_id` as a job parameter. The producer job then knows which shared table to read.
- Decoupled — consumer just writes locally and signals
- No cross-workspace write permissions needed
- Producer controls validation logic centrally
- Consumer gets feedback via a status table the producer writes back to a shared acknowledgment view


A few Unity Catalog housekeeping points worth flagging as you design this:
Use service principals (not PATs tied to users) for all cross-workspace job triggers; store their OAuth credentials in Databricks Secrets. The `CAN_TRIGGER` permission on Jobs is workspace-local — you'll need to grant it per consumer workspace, but it's a one-time configuration per consumer onboarding. Delta Sharing across workspaces on the same UC metastore uses internal sharing (no external recipient overhead), which simplifies credential management significantly.

 

 

LR

View solution in original post