Best Practices for Reusable Workflows & Cluster Management Across Repos.

akeel-rehman — Thu, 20 Mar 2025 14:05:10 GMT

Hi everyone,

I am looking for best practices around reusable workflows in Databricks, particularly in these areas:

Reusable Workflows Instead of Repetition: How can we define reusable workflows rather than repeating the same steps across multiple jobs?
Calling Workflows Across Repositories: Is it possible to trigger workflows across different repositories inside an organization? If so, what’s the best way to pass arguments between them?
Cluster Management Optimization: Instead of using separate job clusters for each workflow, how can we optimize cluster management to avoid the overhead of starting and stopping clusters for each job?

Any insights, best practices, or examples would be greatly appreciated! Thanks in advance.

Re: Best Practices for Reusable Workflows & Cluster Management Across Repos.

AbhaySingh — Wed, 29 Oct 2025 09:10:13 GMT

Here are my recommendations:

1. Databricks Asset Bundles (DABs) for reusable workflows

2. API-based triggering and Run Job Tasks for cross-repo workflows

3. Instance Pools as the #1 game-changer for cluster optimization (5-10 seconds vs 5-10 minutes startup)