Cannot run ./Includes/Classroom-Setup-01.1 in Advanced Data Engineering with Databricks with 12

dwalsh
New Contributor III

‎08-20-2024 05:06 PM

I am started the Advanced Data Engineering with Databricks course and have tried to run the includes code at the start. We recently had issues with 12.2 and moved to a newer version as there appeared to be some issues around setuptools. 

 

If I run "

%run ./Includes/Classroom-Setup-01.1

" using 12.2 I get the following

java.lang.Exception: Process List(virtualenv, /local_disk0/.ephemeral_nfs/envs/pythonEnv-e9611c15-151b-41e4-9cb5-72c4670cf083, -p, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python, --no-download, --no-setuptools, --no-wheel) exited with code 1.

 

%pwd also returns the same error

If I try to use 14.3 which we currently use in work, it will start running but eventually stops due to not having the version suggested as below.

 

AssertionError: The Databricks Runtime is expected to be one of ['12.2.x-scala2.12', '12.2.x-photon-scala2.12', '12.2.x-cpu-ml-scala2.12'], found "14.3.x-scala2.12". Please see the "Troubleshooting | Spark Version" section of the "Version Info" notebook for more information.
 
 
I'm wondering if anyone has any suggestions on how to proceed here?
0 Kudos