-werners-
Esteemed Contributor III

I see 2 easy ways:

  1. use databricks workflows. this creates job clusters for you
  2. use data factory and the Databricks notebook activity: this does the same

Basically you tell databricks to create short-lived clusters which are alive during the execution of a spark program. When the program is finished, the cluster is terminated.

There is also something like cluster pools, which keeps instances warm.