Demo notebook version and the current databricks don't align

[vasudevkillada]

Hello folks - 
Did anyone come across this problem where the demo notebooks ask you to run - ['13.2.x-scala2.12', '13.2.x-photon-scala2.12', '13.2.x-cpu-ml-scala2.12'] and the databricks version doesn't have anything below found "13.3.x-photon-scala2.12". 

I modified by runtime config from latest to 13.x version but I still get photon-scala error. while I post this I'll disable Photon and try to run to see if I get the notebook executed?
btw I am running databricks acadmedy training 1.8 working with demo notebooks as part of data engineering module 2

[vasudevkillada]

I still get the above error 😞

[vasudevkillada]

I changed the runtime version to 12.x but still same result

[holly]

Those cluster runtimes are quite old and won't appear in the UI. You'll have to change the runtime with the cluster API https://docs.databricks.com/api/workspace/clusters/edit

If you've never done this before, this is some (bad) code that will help

%python

import requests

# You'll need an access token from the developer settings. NEVER STORE IT IN A NOTEBOOK. 
AUTH_HEADER = {"Authorization" : "Bearer " + "dapi0000000000000000000000"}

#will be slightly different if you're in azure
URL = "https:// <your workspace URL> .cloud.databricks.com/api/2.1/clusters/edit"

REQUEST = {"cluster_id": " <Cluster ID found in URL of cluster page> ", "num_workers": 2, "spark_version": "13.2.x-scala2.12","node_type_id": "i3.2xlarge"}

response = requests.post(URL, params=REQUEST, headers=AUTH_HEADER)

# should return a 200
response

[vasudevkillada]

Thank you kindly for responding. Actually I fixed the runtime version issue before this response. Even then it didn't work because I get data catalog error even though I have a metastore in the AWS S3 location - ELECT CURRENT_METASTORE(); returned a result of aws:us-west-2:432ad77b-3.xxxxxx

AssertionError: Failed to create the catalog "vasudev_killada_oiv6_da". Please see the "Troubleshooting | Cannot Create Catalog" section of the "Version Info" notebook for more information.

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-2910bec1-6d83-487b-85ba-d530893816bd/lib/python3.11/site-packages/dbacademy/dbhelper/dbacademy_helper_class.py:473, in DBAcademyHelper.__create_catalog(self) 472 print(f"Creating & using the catalog \"{self.catalog_name}\"", end="...") --> 473 dbgems.sql(f"CREATE CATALOG IF NOT EXISTS {self.catalog_name}") 474 dbgems.sql(f"USE CATALOG {self.catalog_name}")

then I try to validate by creating a catalog and then I get issue with this error:

from pyspark.sql import SparkSession

Create a Spark session

spark = SparkSession.builder.getOrCreate()

Define a unique catalog name

catalog_name = "vasu_catalog_test"  # Ensure this is unique

Try creating and using the catalog

try:

    spark.sql(f"CREATE CATALOG IF NOT EXISTS {catalog_name}")

    spark.sql(f"USE CATALOG {catalog_name}")

    print(f"Successfully created and switched to catalog: {catalog_name}")

except Exception as e:

    print(f"Error creating catalog: {str(e)}")

    raise AssertionError(f"Failed to create the catalog '{catalog_name}'. Please see the 'Troubleshooting | Cannot Create Catalog' section of the 'Version Info' notebook for more information.")