topic Re: Databricks academy error setup - Free Edition with Serverless Compute in Data Engineering

Databricks academy error setup - Free Edition with Serverless Compute

iFoxz17 — Wed, 03 Dec 2025 17:47:25 GMT

Databricks is passing from the Community Edition to the Free Edition, which I am currently using.

When executing the Includes/Classroom-setup notebooks the following exception is raised: [CONFIG_NOT_AVAILABLE] Configuration dbacademy.deprecation.logging is not available. SQLSTATE: 42K0I

Inspecting the code, the problem seems to be related to the spark.conf.get() method, which is declared as follows in the documentation of the academy repo

---------------------------------------------------------

get(self, key: str, default: Union[str, NoneType, pyspark._globals._NoValueType] = <no value>) -> Optional[str] Parameters

key : str | key of the configuration to get.

default : str, optional | value of the configuration to get if the key does not exist.

Returns The string value of the configuration set, or None.

Examples : spark.conf.get("non-existent-key", "my_default") // 'my_default'

--------------------------------------------------------

However, testing the method clearly shows that it is raising an exception instead of returning the default value for the missing key.

This seems to be solvable using spark.conf.getAll.get(key, default), where spark.conf.getAll is retuning a python dict containing all the configurations. However, this problem is present also in the inner code of the "v3.0.23" dbacademy library version I am using, which raises the [CONFIG_NOT_AVAILABLE] error when importing modules of the library (in my case the dbgems module).

Re: Databricks academy error setup - Free Edition with Serverless Compute

ManojkMohan — Wed, 03 Dec 2025 20:44:39 GMT

@iFoxz17 The CONFIG_NOT_AVAILABLE error (SQLSTATE: 42K0I) occurs because spark.conf.get(key) in Databricks Free Edition throws an exception for missing configuration keys like dbacademy.deprecation.logging, instead of returning the default value as expected from PySpark documentation

Solution thinking:

Use spark.conf.getAll(): Replace spark.conf.get(key, default) with dict(spark.conf.getAll()).get(key, default) to safely retrieve configs without exceptions, as your testing confirmed
Example: version = dict(spark.conf.getAll()).get("dbacademy.deprecation.logging", "default_value")
Pre-set configurations: Add spark.conf.set("dbacademy.deprecation.logging", "false") (or appropriate value) at the notebook start before imports https://community.databricks.com/t5/data-engineering/databricks-academy-setup-error-data-engineering/td-p/90844
Switch to DBR 13.3 LTS if available in Free Edition for better compatibility
Check for newer dbacademy versions via %pip install --upgrade dbacademy Check for newer dbacademy versions via %pip install --upgrade dbacademy

Re: Databricks academy error setup - Free Edition with Serverless Compute

iFoxz17 — Thu, 04 Dec 2025 08:59:10 GMT

@ManojkMohan as mentioned in the first post I already used dict(spark.conf.getAll()).get(key, default) where possible.
However the problem stands when importing modules, like:

- from dbacademy import dbgems

- from dbacademy.dbhelper import DBAcademyHelper, Paths, CourseConfig, LessonConfig

- from dbacademy.dbhelper.validations.validation_helper_class import ValidationHelper

I also tried to use the last release (Release v5.0.11 · databricks-academy/dbacademy), without success.

Finally, the idea of setting manually the missing keys with spark.conf.set is failing too, raising the same exception for the missing key:
--------------------------------------------------------------

spark.conf.set("missing_key", "default")

[CONFIG_NOT_AVAILABLE] Configuration missing_key is not available. SQLSTATE: 42K0I

--------------------------------------------------------------

Re: Databricks academy error setup - Free Edition with Serverless Compute

Hubert-Dudek — Thu, 04 Dec 2025 09:45:37 GMT

Free edition doesn't have DBR versioning as it is serverless, doesn't allow spark.conf.set etc...