Scheduling and triggering jobs based on time and frequency precedence
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-29-2025 09:08 AM
I have a table in Databricks that stores job information, including fields such as job_name, job_id, frequency, scheduled_time, and last_run_time.
I want to run a query every 10 minutes that checks this table and triggers a job if the scheduled_time is less than or equal to the current time.
Some jobs have multiple frequencies, for example, the same job might run daily and monthly. In such cases, I want the lower-frequency job (e.g., monthly) to take precedence, meaning only the monthly job should trigger and the higher-frequency job (daily) should be skipped when both are due.
What is the best way to implement this scheduling and job-triggering logic in Databricks?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-29-2025 11:14 AM
You could add a job with a scheduled based trigger that runs every 10 minutes. The task at the start of the job runs a SQL query against the job information table and uses the logic you described above to output a boolean value. Then feed that boolean value into a conditional task and if True then have a job task that executes your main pipeline.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-29-2025 12:11 PM
Yeah, that’s basically what I’m doing now, but it’s becoming pretty difficult to manage since there are around 120 pipelines in total. Maintaining a separate scheduled job and conditional logic for each one adds a lot of overhead, and it’s getting hard to scale or keep track of everything. That’s why I was exploring whether there’s a cleaner way to handle this kind of multi-schedule orchestration directly.
