topic Databricks workflow with sequenced tasks in Get Started Discussions

Databricks workflow with sequenced tasks

h2p5cq8 — Fri, 10 Jan 2025 17:09:13 GMT

I have a continuous workflow. It is continuous because I would like it to run every minute and if it has stuff to do the first task will take several minutes. As I understand, continuous workflows won't requeue while a job is currently running, whereas scheduled/periodic workflows will be queued if a previous job is running. My problem is that I need to add a second task in the workflow that runs after the first task. When I try to create the second task I get an error saying "having dependencies inside a task is not allowed in continuous job." The dependency here is that the second task comes after the first task. How can I create a continuous workflow with multiple, sequenced tasks? If this can't be done, how can I have a frequently scheduled workflow that does not queue a second job while a previous job is still running? Thank you!

Re: Databricks workflow with sequenced tasks

Alberto_Umana — Fri, 10 Jan 2025 17:17:46 GMT

Hi @h2p5cq8,

Unfortunately, continuous workflows typically don’t support dependencies between tasks, as they are designed to run continuously without a defined start and end. Let me check for additional approaches.

Re: Databricks workflow with sequenced tasks

Alberto_Umana — Fri, 10 Jan 2025 17:32:20 GMT

Hi @h2p5cq8,

Is it possible for you to Instead of a continuous workflow, you can use a scheduled workflow that runs every minute. To prevent multiple instances from running simultaneously, you can implement concurrency control:

1. Set the workflow to run every minute using a cron expression like `* * * * *`.

2. At the beginning of your workflow, add a check to see if a previous instance is still running.

Or add dependency tasks in your workflow.

Re: Databricks workflow with sequenced tasks

h2p5cq8 — Fri, 10 Jan 2025 17:46:26 GMT

Thank you @Alberto_Umana. I am happy to make it a scheduled workflow that runs every minute. I have set concurrency to 1, but will that keep subsequent jobs from queueing up? If not, is there some other way to stop Databricks from queueing a subsequent job while a previous job is running? I don't think it's absolutely required; I am just trying to avoid a scenario where a job becomes long running and the job queue starts filling up.

Re: Databricks workflow with sequenced tasks

Alberto_Umana — Fri, 10 Jan 2025 18:05:53 GMT

Hi @h2p5cq8,

No problem! and you can have the queue option disabled to stop it. Go to the Advanced settings in the Job details side panel and toggle off the Queue option to prevent jobs from being queued

Re: Databricks workflow with sequenced tasks

h2p5cq8 — Fri, 10 Jan 2025 18:28:59 GMT

Perfect! Thank you for your help @Alberto_Umana