cancel
Showing results for 
Search instead for 
Did you mean: 
Training offerings
Explore discussions on Databricks training programs and offerings within the Community. Get insights, recommendations, and support from peers to maximize your learning experience and advance your skills in data analytics and machine learning.
cancel
Showing results for 
Search instead for 
Did you mean: 

Demo notebook version and the current databricks don't align

vasudevkillada
New Contributor III

Hello folks - 
Did anyone come across this problem where the demo notebooks ask you to run - ['13.2.x-scala2.12', '13.2.x-photon-scala2.12', '13.2.x-cpu-ml-scala2.12'] and the databricks version doesn't have anything below found "13.3.x-photon-scala2.12". 

I modified by runtime config from latest to 13.x version but I still get photon-scala error. while I post this I'll disable Photon and try to run to see if I get the notebook executed?
btw I am running databricks acadmedy training 1.8 working with demo notebooks as part of data engineering module 2

 

vasudevkillada_0-1722352952802.png

 

4 REPLIES 4

vasudevkillada
New Contributor III

vasudevkillada_0-1722353787905.png

I still get the above error 😞

 

 

 

vasudevkillada
New Contributor III

I changed the runtime version to 12.x but still same result

vasudevkillada_1-1722355123169.png

 

holly
Valued Contributor III

Those cluster runtimes are quite old and won't appear in the UI. You'll have to change the runtime with the cluster API https://docs.databricks.com/api/workspace/clusters/edit

If you've never done this before, this is some (bad) code that will help

%python

import requests

# You'll need an access token from the developer settings. NEVER STORE IT IN A NOTEBOOK.
AUTH_HEADER = {"Authorization" : "Bearer " + "dapi0000000000000000000000"}

#will be slightly different if you're in azure
URL = "https:// <your workspace URL> .cloud.databricks.com/api/2.1/clusters/edit"

REQUEST = {"cluster_id": " <Cluster ID found in URL of cluster page> ", "num_workers": 2, "spark_version": "13.2.x-scala2.12","node_type_id": "i3.2xlarge"}

response = requests.post(URL, params=REQUEST, headers=AUTH_HEADER)

# should return a 200
response

vasudevkillada
New Contributor III

Thank you kindly for responding. Actually I fixed the runtime version issue before this response. Even then it didn't work because I get data catalog error even though I have a metastore in the AWS S3 location - ELECT CURRENT_METASTORE(); returned a result of aws:us-west-2:432ad77b-3.xxxxxx

AssertionError: Failed to create the catalog "vasudev_killada_oiv6_da". Please see the "Troubleshooting | Cannot Create Catalog" section of the "Version Info" notebook for more information.
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-2910bec1-6d83-487b-85ba-d530893816bd/lib/python3.11/site-packages/dbacademy/dbhelper/dbacademy_helper_class.py:473, in DBAcademyHelper.__create_catalog(self) 472 print(f"Creating & using the catalog \"{self.catalog_name}\"", end="...") --> 473 dbgems.sql(f"CREATE CATALOG IF NOT EXISTS {self.catalog_name}") 474 dbgems.sql(f"USE CATALOG {self.catalog_name}")

then I try to validate by creating a catalog and then I get issue with this error:
from pyspark.sql import SparkSession

Create a Spark session
spark = SparkSession.builder.getOrCreate()

Define a unique catalog name
catalog_name = "vasu_catalog_test"  # Ensure this is unique

Try creating and using the catalog
try:
    spark.sql(f"CREATE CATALOG IF NOT EXISTS {catalog_name}")
    spark.sql(f"USE CATALOG {catalog_name}")
    print(f"Successfully created and switched to catalog: {catalog_name}")
except Exception as e:
    print(f"Error creating catalog: {str(e)}")
    raise AssertionError(f"Failed to create the catalog '{catalog_name}'. Please see the 'Troubleshooting | Cannot Create Catalog' section of the 'Version Info' notebook for more information.")

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group