Hi Databricks Community,

If only possible I would like to use Shared Jobs Cluster on external orchestrator like Azure Data Factory (ADF) or Synapse Workspace.
The main reasons for using Shared Job cluster are:

• reduction of start-up time (<1min vs 5 min per activity)
• reduction of compute cost for the underlying vm
• possibly reuse / caching some data

In other words if we have multiple databricks activities being run in sequence for the same data (common practice in medallion architecture) we would like avoid situation that we treat each of them as completely isolated runs.
It is possible in databricks workflows: 
https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/use-compute#use-shared-job-cluster...
How to Save Time and Costs With Cluster Reuse in Databricks Jobs - The Databricks Blog

Is it possible to use this feature in the external orchestrator like ADF?

I would like to avoid creating custom synchronisation using workflows triggered and status checked via REST API as described here: 
Leverage Azure Databricks jobs orchestration from Azure Data Factory - Microsoft Community Hub
or here: 
How to orchestrate Databricks jobs from Azure Data Factory using Databricks REST API | Medium

The native databricks ADF connector in my view is almost always the best option. Please consider fact that due to other requirements I am not able to use workflows directly.
I would imagine that by supplying a common attribute, like a 'pipeline().RunId' + 'WaitForNext' flag one could reuse existing cluster