best alternative to webhooks with Unity Catalog?

bragoarefur — Tue, 05 Nov 2024 00:36:05 GMT

We have been using the feature MLflow Model Registry Webhooks on Azure Databricks . Webhooks are in public preview, but oddly are deprecated at the same time with the introduction of Unity Catalog. The documentation says "For an alternative, see Can I use stage transition requests or trigger webhooks on events?." but that page does not in fact describe a real alternative. Using webhooks allows us to hook specific events like MODEL_VERSION_CREATED. There doesn't appear to be any equivalent feature in Unity Catalog (UC).

After some digging, I see two main alternatives:

Run a periodic job that polls UC for new model versions
Use audit log streaming to detect new model versions

Recommendations? Option #1 is simpler but polling is inefficient and introduces latency. Option #2 is more elegant and scalable but is also complex. For example, audit logging must be turned on; raw audit logs have to go somewhere; Unity Catalog needs to load those audit logs; audit log storage is not cloud-agnostic.

Sad to see webhooks going away and hope that Databricks will create a similar capability for Unity Catalog.

Re: best alternative to webhooks with Unity Catalog?

Louis_Frolio — Thu, 30 Oct 2025 10:03:56 GMT

Greetings @bragoarefur , Thanks for laying out the context so clearly—your read of the current state is spot on. Below is what the Databricks docs actually say, and a recommendation with an implementation path that avoids polling while staying cloud-agnostic.

What the docs say

Workspace Model Registry webhooks are in Public Preview and let you trigger on events such as MODEL_VERSION_CREATED, but they are explicitly not available when using Models in Unity Catalog (UC).
The UC migration guide’s “manual approval” alternative effectively suggests using job notifications to call out to external CI/CD systems for approvals, rather than event-driven registry webhooks; it does not provide an equivalent eventing facility for UC models.
Models in Unity Catalog do not have an activity log in the UI; tracking activity requires using audit logs.

Viable alternatives in UC

You identified the two main paths; here’s how they stack up against the platform’s capabilities:

Option #1: Poll UC for new model versions. This works via MLflow client calls (for example, listing model versions), but there’s no native activity log to subscribe to, so you end up implementing your own state tracking and timers. It’s simple, but inefficient and introduces latency and drift.
Option #2: Consume Databricks audit logs (System Tables).
- UC emits audit events for model lifecycle actions, including createModelVersion, finalizeModelVersion, getModelVersion, updateModelVersion, deleteModelVersion, and setRegisteredModelAlias, among others—exactly the signals you need to detect new versions and promotions.
- These are available via the system.access.audit system table (Public Preview) governed by UC, with a stable schema and retention, and they can be consumed with streaming or scheduled queries. This is Databricks-hosted and consistent across clouds, removing the need to set up cloud-specific storage to receive logs.
- Note that system tables are updated throughout the day (not true real-time). If you need sub-minute latencies, you may still want a narrow, model-specific overlay (e.g., short-interval polling) to bridge the gap for critical models.

Recommendation

Given your requirements and trade-offs, I recommend Option #2—use audit log System Tables as the backbone, with a small overlay for truly latency-sensitive models:

Use system.access.audit to detect UC model events (e.g., createModelVersion, finalizeModelVersion, setRegisteredModelAlias). It’s scalable, robust, and cloud-agnostic from your perspective because Databricks hosts the storage and shares it via Delta Sharing across clouds.
For the small subset of models where sub-minute reaction time matters, add a targeted polling overlay (MLflow client list/search) with short intervals. Keep this scoped to only those models to avoid system-wide inefficiency. The general model activity is not exposed via a UC activity log, so polling would be the only way to get that near-real-time signal.
If you still require a manual approval step, use job notifications to fan out to your external CI/CD systems and gate deployment there; this is the UC guide’s recommended approach for manual approvals post-train.

Reference implementation sketch (System Tables)

You can stream from the audit system table and trigger downstream jobs or notifications when specific UC model events arrive. Example in PySpark:

from pyspark.sql.functions import col

# Stream audit logs
audit_stream = (
    spark.readStream.table("system.access.audit")
    .where(col("service_name") == "unityCatalog")
    .where(col("action_name").isin(
        "createModelVersion",
        "finalizeModelVersion",
        "setRegisteredModelAlias",
        "updateModelVersion",
        "deleteModelVersion"
    ))
)

# Write matching events to a Bronze table or directly to a trigger function
query = (
    audit_stream.writeStream
    .option("checkpointLocation", "dbfs:/checkpoints/uc_model_events")
    .toTable("prod.ml_ops.uc_model_events_bronze")
)

And a simple SQL view over system.access.audit for scheduled detection:

CREATE OR REPLACE VIEW prod.ml_ops.uc_model_events AS
SELECT
  event_time,
  action_name,
  request_params,
  user_identity.email AS actor
FROM system.access.audit
WHERE service_name = 'unityCatalog'
  AND action_name IN (
    'createModelVersion',
    'finalizeModelVersion',
    'setRegisteredModelAlias',
    'updateModelVersion',
    'deleteModelVersion'
  );

Operational notes:

Ensure your workspace is enabled for Unity Catalog and that metastore/account admins grant USE/SELECT on the system schemas so your pipelines can read system tables.
System table data is Databricks-hosted in the same region as your metastore; you don’t need to configure S3/ABFS/GCS delivery to consume these via SQL/streaming, which addresses cloud-specific storage concerns.
Expect “updated throughout the day” semantics rather than true real-time; tune trigger intervals and watermarking accordingly, and consider the hybrid overlay for the smallest set of latency-sensitive models.

What’s missing today

There is currently no webhook equivalent for UC models—the docs explicitly state UC models do not support registry webhooks.
For manual approval, the recommended path is post-train job notifications to external CI/CD rather than registry webhooks.

Bottom line

Use the audit log system.access.audit table for event-driven workflows on UC models, and reserve polling for the few cases where ultra-low latency is critical. This keeps the architecture elegant, scalable, and cloud-agnostic, while acknowledging the current absence of UC webhooks.

Hope this helps, Louis.

topic Re: best alternative to webhooks with Unity Catalog? in Data Governance