Hi there,
I'm curious if anyone is able to definitively help me answer how DLT Job Clusters operate/run.
For example, the following is my baseline understanding of DLT Job Clusters. If I run a Triggered DLT Pipeline (e.g. daily) the job cluster takes more or less 5 minutes to spin up until it can start running through the operations. If I have it in Development mode the job cluster stays alive for a bit longer for testing/development, but if it's in Production mode the cluster starts and is killed immediately.
I'm curious about the for a "Semi-Continuous" (e.g. running every minute) or a Continuous Job Cluster, what happens to these DLT Job Clusters? Are they "Always On" in either/both of these cases? For the one that runs every minute, does it spin up an entire job cluster every time (which takes usually about 5 minutes)??