Agree with previous responses BUT consider using a dedicated cluster if using it under an "automation security principal" identity for a subset of your jobs, if that makes sense. To summarize, you need to run a job, under a non-human identity and you need an active cluster, not possible to wait for creating a jobs compute cluster (it takes minutes despite using pools) and serverless compute is non an option for you (maybe for security or other reasons it's not enabled yet in your company).