Hi, thank you for your response! Yes, I did cut off the yaml snippet to the problem area since the entire yaml is quite a large file. We do define the job-cluster in the workflow yaml. "job-cluster" is just a pseudonym. Sorry for the confusing snippet. Here is the portion where the job cluster is defined:

job_clusters:
        - job_cluster_key: job-cluster
          new_cluster:
            spark_version: 12.2.x-scala2.12
            node_type_id: m5d.4xlarge
            driver_node_type_id: m5d.4xlarge
            data_security_mode: SINGLE_USER
            runtime_engine: PHOTON
            autoscale:
              min_workers: 2
              max_workers: 12
            aws_attributes:
              instance_profile_arn: [removed]
              zone_id: auto
              first_on_demand: 1