Sharing compute between tasks of a job

[mvmiller]

Is there a way to set up a workflow with multiple tasks, so that different tasks can share the same compute resource, at the same time?

I understand that an instance pool may be an option, here. Wasn't sure if there were other possible options to consider, though.

[Kaniz_Fatma]

Hi @mvmiller , Certainly! When orchestrating workflows with multiple tasks, it’s essential to optimize resource usage.

Here are a couple of approaches you can consider:

GitHub Actions:

• By default, GitHub Actions runs multiple jobs in parallel. However, you can configure them to run sequentially if needed.
• To share setup code across jobs, consider creating a composite action. This way, you define the common setup steps once and reuse them across different jobs.
• Alternatively, you could create your own custom GitHub Action with the shared setup code or use a Docker image pre-installed with the necessary dependencies.
• If you want independent checkmarks for the linter and test suite, you can still achieve that by stru....

Databricks (for more complex scenarios):

• If you’re working with Databricks, you can use shared job clusters to optimize resource usage.
• A shared job cluster allows multiple tasks within the same job run to reuse the cluster. You can either use a single job cluster for all tasks or create multiple job clusters optimized for ....

Remember that the choice depends on your specific requirements and the tools you’re using. Whether it’s GitHub Actions or Databricks, efficient resource utilization is key! 🚀