Hi everyone,
We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" pattern.
A few questions for those further along in this migration:
- How are you using File Arrival Triggers and Table Update Triggers to avoid running compute against empty or unchanged sources? Have you seen a measurable DBU savings from this switch?
- For cross-workspace dependencies (Job A in Workspace 1 needs to complete before Job B in Workspace 2 starts), are you using a "Signal Table" pattern in Unity Catalog, or is there a more native way to handle this in Lakeflow?
- How do you handle "fan-out" dependencies—i.e., one upstream table update needs to trigger 5-6 downstream jobs owned by different teams? Are you managing this centrally, or letting each team own their own trigger subscription?
Would love to hear the "messy" lessons learned, not just the documented happy path.
Data Architect | 13 Years Domain Expertise | Databricks SA Champion Cohort