topic Re: Runtime Spark 13.2 no longer available in Training offerings

Runtime Spark 13.2 no longer available

bedant — Mon, 19 Feb 2024 19:52:20 GMT

I'm enrolled in the course Get Started with Databricks for Data Engineering

Trying work on one of the demos that asked to copy the code base

https://labs.training.databricks.com/import/get-started-with-data-engineering-on-databricks/v2.0.3/get-started-with-data-engineering-on-databricks.dbc?token=yzlmn3ek00&expires_at_utc=2024-02-20T07:35:35

I tried to run the code under Module 2.3 Demo: Data Management, but was not able to run the first code, under

Includes/Classroom-Setup-05, cause it calls the code under /Includes/_common.

That indicates the the DB Spark compute should be 13.2, but Azure no longer has 13.2, it has 13.3 and higher.

How to fix the _common code or any other solution?

Re: Runtime Spark 13.2 no longer available

feiyun0112 — Tue, 20 Feb 2024 07:21:28 GMT

I think you should change _common code to use actual databricks version

course_config = CourseConfig(course_code = "gswdeod", course_name = "get-started-with-data-engineering-on-databricks", data_source_name = "get-started-with-data-engineering-on-databricks", data_source_version = "v01", install_min_time = "1 min", install_max_time = "5 min", remote_files = remote_files, supported_dbrs = ["13.2.x-scala2.12", "13.2.x-photon-scala2.12", "13.2.x-cpu-ml-scala2.12"], expected_dbrs = "13.2.x-scala2.12, 13.2.x-photon-scala2.12, 13.2.x-cpu-ml-scala2.12")

Re: Runtime Spark 13.2 no longer available

bedant — Mon, 26 Feb 2024 17:26:04 GMT

Yes updated to my exact spark version and now it worked 🙂

supported_dbrs = ["13.3.x-scala2.12", "13.3.x-photon-scala2.12", "13.3.x-cpu-ml-scala2.x"],

expected_dbrs = "13.3.x-scala2.12, 13.3.x-photon-scala2.12, 13.3.x-cpu-ml-scala2.x")