Hi Ajay,

This is expected behavior. A job cluster is designed to be ephemeral, so Databricks creates a new cluster for each run and terminates it when the run finishes. Currently, there isn't an option to keep the same job cluster alive across continuous job runs.

If reducing startup time is important, consider using an existing cluster instead of a job cluster. That way, the same cluster can be reused until it's manually stopped or reaches its auto-termination timeout.