Databricks academy error setup - Free Edition with Serverless Compute

[iFoxz17]

Databricks is passing from the Community Edition to the Free Edition, which I am currently using.

When executing the Includes/Classroom-setup notebooks the following exception is raised: [CONFIG_NOT_AVAILABLE] Configuration dbacademy.deprecation.logging is not available. SQLSTATE: 42K0I

Inspecting the code, the problem seems to be related to the spark.conf.get() method, which is declared as follows in the documentation of the academy repo

---------------------------------------------------------

get(self, key: str, default: Union[str, NoneType, pyspark._globals._NoValueType] = <no value>) -> Optional[str] Parameters

key : str | key of the configuration to get.

default : str, optional | value of the configuration to get if the key does not exist. 

Returns The string value of the configuration set, or None. 

Examples : spark.conf.get("non-existent-key", "my_default")  // 'my_default'

--------------------------------------------------------

However, testing the method clearly shows that it is raising an exception instead of returning the default value for the missing key.

This seems to be solvable using spark.conf.getAll.get(key, default), where spark.conf.getAll is retuning a python dict containing all the configurations. However, this problem is present also in the inner code of the "v3.0.23" dbacademy library version I am using, which raises the [CONFIG_NOT_AVAILABLE] error when importing modules of the library (in my case the dbgems module).

[ManojkMohan]

@iFoxz17 The CONFIG_NOT_AVAILABLE error (SQLSTATE: 42K0I) occurs because spark.conf.get(key) in Databricks Free Edition throws an exception for missing configuration keys like dbacademy.deprecation.logging, instead of returning the default value as expected from PySpark documentation

Solution thinking:

• Use spark.conf.getAll(): Replace spark.conf.get(key, default) with dict(spark.conf.getAll()).get(key, default) to safely retrieve configs without exceptions, as your testing confirmed
• Example: version = dict(spark.conf.getAll()).get("dbacademy.deprecation.logging", "default_value")
• Pre-set configurations: Add spark.conf.set("dbacademy.deprecation.logging", "false") (or appropriate value) at the notebook start before imports https://community.databricks.com/t5/data-engineering/databricks-academy-setup-error-data-engineering...
• Switch to DBR 13.3 LTS if available in Free Edition for better compatibility
• Check for newer dbacademy versions via %pip install --upgrade dbacademy Check for newer dbacademy versions via %pip install --upgrade dbacademy

[Hubert-Dudek]

Free edition doesn't have DBR versioning as it is serverless, doesn't allow spark.conf.set etc...

[iFoxz17]

@ManojkMohan as mentioned in the first post I already used dict(spark.conf.getAll()).get(key, default) where possible.
However the problem stands when importing modules, like:

- from dbacademy import dbgems

- from dbacademy.dbhelper import DBAcademyHelper, Paths, CourseConfig, LessonConfig

- from dbacademy.dbhelper.validations.validation_helper_class import ValidationHelper

I also tried to use the last release (Release v5.0.11 · databricks-academy/dbacademy), without success.

Finally, the idea of setting manually the missing keys with spark.conf.set is failing too, raising the same exception for the missing key:
--------------------------------------------------------------

spark.conf.set("missing_key", "default")

[CONFIG_NOT_AVAILABLE] Configuration missing_key is not available. SQLSTATE: 42K0I

--------------------------------------------------------------