Rearrange tasks in databricks workflow

[chemajar]

Hello,

There is anyway to rearrange tasks in databricks workflow?.

I would like that line that join the two marked tasks doesn't pass behind the other tasks.  It is posible that this line by one side?

Thanks.

[Kaniz_Fatma · Accepted solution]

Hi @chemajar,  In Databricks Workflows, you can arrange tasks and control their execution order.

Let’s explore how you can achieve this:

1. Task Dependencies:

   • By default, a task runs when its dependencies have executed successfully. You can configure tasks to run only when specific conditions are met¹.
   • To set task configuration options:
     • Click on Workflows in the sidebar.
     • Locate the job name and click on it.
     • Navigate to the Tasks tab and select the task you want to edit².

2. Task Ordering:

   • If you want to ensure that the line connecting two marked tasks doesn’t pass behind other tasks, consider the following approaches:
     • Swap Tasks:
       • Navigate to the Data Science & Engineering homepage.
       • Click on Workflows.
       • Select a job name and find the compute in the left panel.
       • Click the Swap button.
       • Choose an existing job cluster (if available) or create a new one³.
     • Manual Ordering:
       • Arrange the tasks in the desired order within your workflow.
       • Ensure that the dependent tasks are placed before the tasks they rely on.

3. Visual Clarity:

   • While Databricks doesn’t provide a direct way to move the connecting line to one side, you can enhance visual clarity by organizing tasks logically.
   • Consider grouping related tasks together and placing them in a sequence that makes sense for your workflow.

Remember that Databricks Jobs determines whether a job run was successful based on the outcome of leaf tasks (tasks with no downstream dependencies). So, ensure that your task dependencies are correctly defined to achieve the desired execution order¹.

Feel free to explore these options and adapt them to your specific workflow needs! 🚀

[chemajar · Accepted solution]

Thank you very much, the Visual Clarity part was what I thoght

[artsheiko]

Hi @chemajar,

Take a look at Databricks Asset Bundles. It allows you to streamline the development of complex workflows using a yaml definition. In case you need to change the task dependencies, you can rearrange the flow as you need just change the yaml definition.

If you need a yaml definition for an existing workflow, you can find it by clicking on a three-points menu at the top right corner of the chosen workflow.