Best Practices for Reusable Workflows & Cluster Management Across Repos.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-20-2025 07:05 AM
Hi everyone,
I am looking for best practices around reusable workflows in Databricks, particularly in these areas:
- Reusable Workflows Instead of Repetition: How can we define reusable workflows rather than repeating the same steps across multiple jobs?
- Calling Workflows Across Repositories: Is it possible to trigger workflows across different repositories inside an organization? If so, what’s the best way to pass arguments between them?
- Cluster Management Optimization: Instead of using separate job clusters for each workflow, how can we optimize cluster management to avoid the overhead of starting and stopping clusters for each job?
Any insights, best practices, or examples would be greatly appreciated! Thanks in advance.
Simplicity & Togetherness
Labels:
- Labels:
-
Workflows
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-29-2025 02:10 AM
Here are my recommendations:
1. Databricks Asset Bundles (DABs) for reusable workflows
2. API-based triggering and Run Job Tasks for cross-repo workflows
3. Instance Pools as the #1 game-changer for cluster optimization (5-10 seconds vs 5-10 minutes startup)