I am started the Advanced Data Engineering with Databricks course and have tried to run the includes code at the start. We recently had issues with 12.2 and moved to a newer version as there appeared to be some issues around setuptools.
If I run "
%run ./Includes/Classroom-Setup-01.1
" using 12.2 I get the following
java.lang.Exception: Process List(virtualenv, /local_disk0/.ephemeral_nfs/envs/pythonEnv-e9611c15-151b-41e4-9cb5-72c4670cf083, -p, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python, --no-download, --no-setuptools, --no-wheel) exited with code 1.
%pwd also returns the same error
If I try to use 14.3 which we currently use in work, it will start running but eventually stops due to not having the version suggested as below.
AssertionError: The Databricks Runtime is expected to be one of ['12.2.x-scala2.12', '12.2.x-photon-scala2.12', '12.2.x-cpu-ml-scala2.12'], found "14.3.x-scala2.12". Please see the "Troubleshooting | Spark Version" section of the "Version Info" notebook for more information.
I'm wondering if anyone has any suggestions on how to proceed here?
