Best Practices for Reusable Workflows & Cluster Management Across Repos.

[akeel-rehman]

Hi everyone,

I am looking for best practices around reusable workflows in Databricks, particularly in these areas:

1. Reusable Workflows Instead of Repetition: How can we define reusable workflows rather than repeating the same steps across multiple jobs?
2. Calling Workflows Across Repositories: Is it possible to trigger workflows across different repositories inside an organization? If so, what’s the best way to pass arguments between them?
3. Cluster Management Optimization: Instead of using separate job clusters for each workflow, how can we optimize cluster management to avoid the overhead of starting and stopping clusters for each job?  

Any insights, best practices, or examples would be greatly appreciated! Thanks in advance.