Demo notebook version and the current databricks don't align

vasudevkillada
New Contributor III

‎07-30-2024 08:26 AM

Hello folks - 
Did anyone come across this problem where the demo notebooks ask you to run - ['13.2.x-scala2.12', '13.2.x-photon-scala2.12', '13.2.x-cpu-ml-scala2.12'] and the databricks version doesn't have anything below found "13.3.x-photon-scala2.12". 

I modified by runtime config from latest to 13.x version but I still get photon-scala error. while I post this I'll disable Photon and try to run to see if I get the notebook executed?
btw I am running databricks acadmedy training 1.8 working with demo notebooks as part of data engineering module 2

 

vasudevkillada_0-1722352952802.png

 

0 Kudos

vasudevkillada
New Contributor III

‎07-30-2024 08:36 AM

vasudevkillada_0-1722353787905.png

I still get the above error 😞

 

 

 

0 Kudos

vasudevkillada
New Contributor III

‎07-30-2024 08:59 AM

I changed the runtime version to 12.x but still same result

vasudevkillada_1-1722355123169.png

 

0 Kudos

holly
Databricks Employee
Databricks Employee

‎07-31-2024 06:11 AM

Those cluster runtimes are quite old and won't appear in the UI. You'll have to change the runtime with the cluster API https://docs.databricks.com/api/workspace/clusters/edit

If you've never done this before, this is some (bad) code that will help

%python

import requests

# You'll need an access token from the developer settings. NEVER STORE IT IN A NOTEBOOK. 
AUTH_HEADER = {"Authorization" : "Bearer " + "dapi0000000000000000000000"}

#will be slightly different if you're in azure
URL = "https:// <your workspace URL> .cloud.databricks.com/api/2.1/clusters/edit"

REQUEST = {"cluster_id": " <Cluster ID found in URL of cluster page> ", "num_workers": 2, "spark_version": "13.2.x-scala2.12","node_type_id": "i3.2xlarge"}

response = requests.post(URL, params=REQUEST, headers=AUTH_HEADER)

# should return a 200
response

vasudevkillada
New Contributor III
In response to holly

‎08-01-2024 03:12 PM

Thank you kindly for responding. Actually I fixed the runtime version issue before this response. Even then it didn't work because I get data catalog error even though I have a metastore in the AWS S3 location - ELECT CURRENT_METASTORE(); returned a result of aws:us-west-2:432ad77b-3.xxxxxx

AssertionError: Failed to create the catalog "vasudev_killada_oiv6_da". Please see the "Troubleshooting | Cannot Create Catalog" section of the "Version Info" notebook for more information.
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-2910bec1-6d83-487b-85ba-d530893816bd/lib/python3.11/site-packages/dbacademy/dbhelper/dbacademy_helper_class.py:473, in DBAcademyHelper.__create_catalog(self) 472 print(f"Creating & using the catalog \"{self.catalog_name}\"", end="...") --> 473 dbgems.sql(f"CREATE CATALOG IF NOT EXISTS {self.catalog_name}") 474 dbgems.sql(f"USE CATALOG {self.catalog_name}")

then I try to validate by creating a catalog and then I get issue with this error:
from pyspark.sql import SparkSession

Create a Spark session
spark = SparkSession.builder.getOrCreate()

Define a unique catalog name
catalog_name = "vasu_catalog_test"  # Ensure this is unique

Try creating and using the catalog
try:
    spark.sql(f"CREATE CATALOG IF NOT EXISTS {catalog_name}")
    spark.sql(f"USE CATALOG {catalog_name}")
    print(f"Successfully created and switched to catalog: {catalog_name}")
except Exception as e:
    print(f"Error creating catalog: {str(e)}")
    raise AssertionError(f"Failed to create the catalog '{catalog_name}'. Please see the 'Troubleshooting | Cannot Create Catalog' section of the 'Version Info' notebook for more information.")

0 Kudos