The last paragraph reinforced issues I was seeing with the expected behavior of cluster config in jobs api vs policy s actual behavior. I was trying to specify the spark version in the cluster policy and omit it from the JobsAPI, this didnt work.

Although I will say it didn't seem like any configurations were passed into api-created-cluster Not even my spark configurations were passed in