cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Time window for "All tables are updated" option in job Table Update Trigger

SamAdams
Contributor

I've been using the Table Update Trigger for some SQL alert workflows. I have a job that uses 3 tables with an "All tables updated" trigger:
Table 1 was updated at 07:20 UTC
Table 2 was updated at 16:48 UTC
Table 3 was updated at 16:50 UTC
-> Job is triggered. Updates happened over a ~10 hour time window.
Is there any sense of how long the window is for all tables in the trigger to have an update?

1 REPLY 1

mark_ott
Databricks Employee
Databricks Employee

There is no fixed or documented โ€œwindowโ€ time for the interval between updates to all monitored tables before a job with an "All tables updated" trigger runs in Databricks. The job is triggered as soon as every table in the set has seen at least one update since the last successful runโ€”regardless of how much time has elapsed between the individual table updates.โ€‹

Timing Behavior

  • If Table 1 is updated at 07:20 UTC, Table 2 at 16:48 UTC, and Table 3 at 16:50 UTC, the trigger waits for the last table in that set to be updated; the job will run after Table 3's update, even though there is a ~10 hour gap between Table 1 and Table 3.โ€‹

  • There is no apparent reset or expiration windowโ€”Databricks tracks updates on all monitored tables since the last job run, and as soon as every table has received an update, the trigger fires.โ€‹

Advanced Timing Options

  • You can optionally set โ€œMinimum time between triggersโ€ (enforces a cooldown between runs irrespective of update activity) and โ€œWait after last changeโ€ (holds off execution for a set period after the most recent update in case additional batch updates occur).โ€‹

  • If both are set, Databricks waits for both timers to expire before firing the trigger.โ€‹

Practical Implication

  • If one table updates hours before others, the job will only trigger once all tables have been updated, even if this takes many hours.โ€‹

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now