topic Re: Spark Version Json code for LTS ML cluster in Machine Learning

Spark Version Json code for LTS ML cluster

confusedIntern — Mon, 11 Jul 2022 19:14:02 GMT

Hi! Any help will be greatly appreciated! Thank you so much!

So I'm currently following this tutorial: https://docs.databricks.com/applications/mlflow/projects.html

In the tutorial, for the cluster specification, it has "spark_version": "7.3.x-scala2.12" in the json code. After I completed the tutorial and run the specified code, I see that the runtime version is 7.3 LTS. I want it to have runtime version 7.3 LTS ML (or specifically 9.1 LTS ML but any LTS ML runtime works). I'm not sure how to accomplish that with the json code.

Please let me know if there is a way! Thank you!

Re: Spark Version Json code for LTS ML cluster

tomasz — Mon, 11 Jul 2022 19:36:47 GMT

You can use the API to get the available runtime versions or the CLI with the command:

databricks clusters spark-versions

Re: Spark Version Json code for LTS ML cluster

tomasz — Mon, 11 Jul 2022 19:47:34 GMT

Also for 9.1 LTS ML it should be: "9.1.x-cpu-ml-scala2.12"

Re: Spark Version Json code for LTS ML cluster

confusedIntern — Mon, 11 Jul 2022 19:48:23 GMT

Thank you so much!

Re: Spark Version Json code for LTS ML cluster

confusedIntern — Mon, 11 Jul 2022 19:50:39 GMT

Yes!! I just figured it out before seeing your post. But thank you so much! I tried it and the cluster is working in 9.1 LTS ML!