We want to create a workflow pipeline in which we trigger a Databricks workflow job from AWS. However, the startup time of Databricks workflow jobs on job compute is over 10 minutes, which is causing issues.

We would like to either avoid this startup time or reduce it to around 1 minute. Is there any way to achieve this? We have tried creating a pool cluster, but there doesn’t seem to be an option to select a pool cluster in the workflow job.

Do we have any suggestions or a better way to optimize this?

We have also tried serverless but due to its limitations it is not useful