Hey @dhruvs2 

You could use Lakeflow Jobs for this. You can add a job as a task:

Then you can just follow the docs from here: https://docs.databricks.com/aws/en/jobs/ there's loads of great sections / tutorials.

To answer your specific question:
When configuring a task you just change the DEPENDS ON:

 And the Run if dependencies

Above, you can see I selected All Succeeded. You can select just All done if you only worry about them completing.

All the best,
BS

hi @BS_THE_ANALYST , thanks for the response.
From what I understand, it seems we’d need to maintain a separate job or pipeline in Databricks to orchestrate everything — is that correct?

Hi @BS_THE_ANALYST ,

How can I ensure that Job C is triggered only after Jobs A and B have completed?

Is it possible for A and B to be in separate workflows with different schedules, and still have C triggered only after both A and B are finished?

Hi @dhruvs2  😀.

A Lakeflow Job consists of tasks. The tasks can be things like notebooks or other jobs. If you want to orchestrate many jobs, I'd agree that having a job to do this is your best bet 😀. Then you can setup the dependencies as you require.

If you get stuck with anything, give me a shout 🙂. 

Once you've got the hang of how the jobs work, you can then look into parameterisation where you can start making things really dynamic! https://docs.databricks.com/aws/en/jobs/job-parameters

Don't forget about monitoring/observability either: https://docs.databricks.com/aws/en/jobs/monitor#view-jobs-and-pipelines 

In terms of compute for running the jobs. I'd say that Serverless is your best bet. If not, and you're using classic compute, it's recommended to use job compute. Here's a good article to read more about compute considerations: https://docs.databricks.com/aws/en/jobs/compute#what-is-the-recommended-compute-for-each-task 

All the best,
BS