cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences?

Khasim_1
New Contributor II

Hi everyone,

We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" pattern.

A few questions for those further along in this migration:

  1. How are you using File Arrival Triggers and Table Update Triggers to avoid running compute against empty or unchanged sources? Have you seen a measurable DBU savings from this switch?
  2. For cross-workspace dependencies (Job A in Workspace 1 needs to complete before Job B in Workspace 2 starts), are you using a "Signal Table" pattern in Unity Catalog, or is there a more native way to handle this in Lakeflow?
  3. How do you handle "fan-out" dependencies—i.e., one upstream table update needs to trigger 5-6 downstream jobs owned by different teams? Are you managing this centrally, or letting each team own their own trigger subscription?

Would love to hear the "messy" lessons learned, not just the documented happy path.

Data Architect | 13 Years Domain Expertise | Databricks SA Champion Cohort
0 REPLIES 0