topic Re: Setup learning environment failed: Configuration dbacademy.library.version is not available. in Data Engineering

Setup learning environment failed: Configuration dbacademy.library.version is not available.

NSJ — Thu, 08 Aug 2024 20:06:44 GMT

Using 1.3 Getting Started with the Databricks Platform Lab. to self learning. When I run DE 2.1 to setup environment, got following error:
Configuration dbacademy.library.version is not available.
Following is the code in the common setup.

specified_version = f"v3.0.70"
key = "dbacademy.library.version"
version = spark.conf.get(key, specified_version)

Which version should I use?

Re: Setup learning environment failed: Configuration dbacademy.library.version is not available.

Luipiu — Thu, 30 Jan 2025 15:47:56 GMT

I resolved adding some instructions in the _common notebook, you can find inside the folder Includes

Put these at the beginning

%pip install git+https://github.com/databricks-academy/dbacademy@v3.0.70

%python dbutils.library.restartPython()

After this I obtain the error for the version of the cluster I was using, so I used the required DBR 11.3 to run the Classroom-Setup script

Re: Setup learning environment failed: Configuration dbacademy.library.version is not available.

Deepa710 — Wed, 05 Feb 2025 07:07:32 GMT

Hi,

I am getting this error after adding pip install and restart in _common.

EnvironmentError: Failed to set environment metadata. The Spark session may be unavailable, please try again or contact Databricks support.

File /databricks/python_shell/dbruntime/PipMagicOverrides.py:129, in PipMagicHandler.runCmd(self, magicCmd, line) 126 self.entry_point.getLogger()._logInfo( 127 f"{__name__}: Setting Setting spark.databricks.connect.environment to {serialized_env}." 128 ) --> 129 spark.conf.set("spark.databricks.connect.environment", serialized_env)

Could you please help?

Regards

Deepa

Re: Setup learning environment failed: Configuration dbacademy.library.version is not available.

Luipiu — Wed, 05 Feb 2025 10:45:13 GMT

Perhaps you haven't the right permission. Are you able to create a SparkSession to check if you have this permission?