topic Re: First notebook in ML course fails with wrong runtime in Data Engineering

First notebook in ML course fails with wrong runtime

IonFreeman_Pace — Wed, 11 Jan 2023 23:18:41 GMT

Help! I'm trying to run this first notebook in the Scalable MachIne LEarning (SMILE) course.

https://github.com/databricks-academy/scalable-machine-learning-with-apache-spark-english/blob/published/ML%2000a%20-%20Spark%20Review.py

It fails on the first cell with

AssertionError: The Databricks Runtime is expected to be one of ['11.3.x-cpu-ml-scala2.12'], found "11.3.x-scala2.12". Please see the "Troubleshooting | Spark Version" section of the "Version Info" notebook for more information.

There's no more information in the Version Info notebook. I set the version to 11.3 and turned off photon to try to get it to match, but I don't see how to get 'cpu-ml' in there.

I suppose I can just modify the 'supported_dbrs' list in Includes/_common, but that seems like it might cause trouble later.

Re: First notebook in ML course fails with wrong runtime

IonFreeman_Pace — Thu, 12 Jan 2023 02:03:39 GMT

It definitely caused trouble later

Re: First notebook in ML course fails with wrong runtime

-werners- — Thu, 12 Jan 2023 10:29:55 GMT

it means your cluster type has to be a ML runtime.

When you create a cluster in databricks, you can choose between different runtimes.

These have different version (spark version), but also different types:

For your case you need to select the ML menu option.

Re: First notebook in ML course fails with wrong runtime

IonFreeman_Pace — Fri, 13 Jan 2023 17:29:01 GMT

Ah, OK, Cool. Thanks for your help. I somehow didn't see that yesterday. I guess you have to know where to look.

Re: First notebook in ML course fails with wrong runtime

Gautamp — Sat, 25 Nov 2023 09:04:13 GMT

was helpful for me too. Thank you for your help during my journey of learning databricks!