Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs

[Khasim_1]

Hi everyone,

We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" pattern.

A few questions for those further along in this migration:

1. How are you using File Arrival Triggers and Table Update Triggers to avoid running compute against empty or unchanged sources? Have you seen a measurable DBU savings from this switch?
2. For cross-workspace dependencies (Job A in Workspace 1 needs to complete before Job B in Workspace 2 starts), are you using a "Signal Table" pattern in Unity Catalog, or is there a more native way to handle this in Lakeflow?
3. How do you handle "fan-out" dependencies—i.e., one upstream table update needs to trigger 5-6 downstream jobs owned by different teams? Are you managing this centrally, or letting each team own their own trigger subscription?

Would love to hear the "messy" lessons learned, not just the documented happy path.

[Sanjeeb2024]

Just answer the details in another thread. Please check.

[Islam_hoti · Accepted solution]

Hi,

Migrating the same way. The messy parts.

On file arrival triggers, first check whether your external locations have file events enabled, because the behaviour differs sharply. Without them you get a cap of 50 triggered jobs per workspace and at most 10,000 files in the monitored path. With file events, the file limit goes away.

Four gotchas that cost us time. Overwriting a file with the same name does not fire the trigger, so any producer writing to a stable filename will silently never run. Conversely, modifying an old file whose metadata has aged out of the internal retention window is treated as a new arrival and fires a spurious run. With file events on, a trigger set on a subpath can time out and error when there is heavy unrelated churn elsewhere under the same root, and the fix is to create a volume mapped to exactly the directory you want and trigger on its root. Worst of all, on S3 and GCS a trigger pointed at a deleted or non existent path never errors and never fires, so you get silence rather than an alert.

Also use the two rate options. Minimum time between triggers is a cooldown, wait after last change is a debounce that resets on each arrival. The second one is what stops a job firing on the first file of a batch of 200.

On savings, frame it carefully. The triggers add no cost beyond cloud listing charges, so the saving is entirely the compute you were spending on runs that found nothing. Measure your current empty run rate before promising a number.

On question two, you likely do not need a signal table. Unity Catalog objects live at the metastore rather than the workspace, so a table update trigger in workspace two can watch a table written from workspace one as long as both share a metastore. The table is the signal. Worth testing how it reacts to optimize and vacuum commits on the watched table, since those are writes too.

On question three, table update triggers are a subscription model, so fan out falls out naturally and the producer does nothing. The messy part is what you lose from Airflow: no single place to see the whole graph. Producers have no idea who depends on them until something breaks, and lineage is the only thing filling that gap.