Robust/complex scheduling with dependency within Databricks?

[RIDBX]

Robust scheduling with dependency within Databricks?

======================================

Thanks for reviewing my threads. I like to explore Robust/complex scheduling with dependency within Databricks.

We know traditional scheduling framework allow robust dependency/conditions setting across multiple tiers etc.

How can we do that within Databricks scheduling?

Eg:

We have a HR application in  tier1 - 100 jobs Start time 12AM

We have a Finance applications in tier2 - 125 jobs Start time 10AM + completion of  HR applications (100 jobs)

These can be run daily or weekly.

How do we do this?

Are there any doc/whitepapers on this subject?

Thanks for your insights.

[pradeep_singh]

Intresting Scenario . This is what i think you can do

Tier 1 job: A single job that contains 100 tasks( where each task trigger a job) scheduled to run at 12:00 AM.

Tier 2 job: A single job that contains 125 tasks, scheduled to run at 10:00 AM.

All 125 tasks in Tier 2 depend on a single SQL Alert task that runs a SQL query against system tables to determine whether the Tier 1 job has completed. As soon as the job completes the SQL Alert task should dependecy would be fullfilled and the job would be ready to run as 10 am when its scheduled to run .

You dont need 100 or 125 task to run each job . You can simply them design by using for each loop and reading parameters from a json for each jobs and if else conditions

[pradeep_singh]

Further readings - 

• SQL Altert Task - https://docs.databricks.com/aws/en/jobs/sql
• If else Task - https://docs.databricks.com/aws/en/jobs/if-else
• For Each task - https://docs.databricks.com/aws/en/jobs/for-each
• Run job task - https://docs.databricks.com/aws/en/jobs/run-job
• Configure task dependencies - https://docs.databricks.com/aws/en/jobs/run-if