topic How do we set spark_version in cluster policies to select the latest GPU ML LTS version as defaultValue? in Data Engineering

How do we set spark_version in cluster policies to select the latest GPU ML LTS version as defaultValue?

Ramana — Mon, 27 Mar 2023 22:13:37 GMT

Currently, I use the below two different JSON snippets to choose either Standard or ML runtime. Similar to the below, what is the defaultValue for spark_version to select the latest GPU ML LTS runtime version?

"spark_version": {

"type": "regex",

"pattern": "[1-9][0-9]\\.[0-9].*",

"defaultValue": "auto:latest-ml"

}

"spark_version": {

"type": "regex",

"pattern": "[1-9][0-9]\\.[0-9].*",

"defaultValue": "auto:latest-lts"

Re: How do we set spark_version in cluster policies to select the latest GPU ML LTS version as defaultValue?

LandanG — Tue, 28 Mar 2023 00:26:15 GMT

Hi @Ramana Kancharana ,

As of right now these options are only available for non-GPU DBRs

Re: How do we set spark_version in cluster policies to select the latest GPU ML LTS version as defaultValue?

Ramana — Tue, 28 Mar 2023 20:06:15 GMT

Thank you. Let me know once it is available.

Re: How do we set spark_version in cluster policies to select the latest GPU ML LTS version as defaultValue?

LandanG — Tue, 28 Mar 2023 20:43:34 GMT

@Ramana Kancharana for sure, will do.