Scheduling and triggering jobs based on time and frequency precedence

[Anoora]

I have a table in Databricks that stores job information, including fields such as job_name, job_id, frequency, scheduled_time, and last_run_time.

I want to run a query every 10 minutes that checks this table and triggers a job if the scheduled_time is less than or equal to the current time.

Some jobs have multiple frequencies, for example, the same job might run daily and monthly. In such cases, I want the lower-frequency job (e.g., monthly) to take precedence, meaning only the monthly job should trigger and the higher-frequency job (daily) should be skipped when both are due.

What is the best way to implement this scheduling and job-triggering logic in Databricks?

[SamAdams]

You could add a job with a scheduled based trigger that runs every 10 minutes. The task at the start of the job runs a SQL query against the job information table and uses the logic you described above to output a boolean value. Then feed that boolean value into a conditional task and if True then have a job task that executes your main pipeline.

https://docs.databricks.com/aws/en/jobs/run-if

[Anoora]

Yeah, that’s basically what I’m doing now, but it’s becoming pretty difficult to manage since there are around 120 pipelines in total. Maintaining a separate scheduled job and conditional logic for each one adds a lot of overhead, and it’s getting hard to scale or keep track of everything. That’s why I was exploring whether there’s a cleaner way to handle this kind of multi-schedule orchestration directly.