best way to store config files in a Unity workspace (Scala/typesafe)

[-werners-]

We use typesafe (scala) to read configuration values from hocon files.
When not using Unity, we read the configuration files from /dbfs/...  works fine.
However, with Unity, usage of dbfs is frowned upon.
So I started looking into alternatives.
And unfortunately, dbfs still seems the best and easiest solution.
Anyone that has a better option which I may have missed?

And no, porting all code to python is not an option 🙂

[Kaniz]

Hi @-werners-, When working with external configuration files on Databricks, consider using the Databricks CLI to securely store and manage your configuration values in environment variables.

You can use the Databricks CLI's secrets command to securely store and manage secret values such as passwords, keys, and tokens in your Databricks workspace. You can reference these private values in your Scala code as environment variables.

[-werners-]

Hi Kaniz, thanks for the suggestion.
However in our case storing config values in env vars will be cumbersome.  We put all of our filters applied in notebooks in those config files, together with default selections for entities etc.
So they are not 'sensitive' values.
I tried using volumes, but unfortunately volumes do not (yet) support scala.io (although it would totally make sense to use volumes here).

[-werners- · Accepted solution]

In the end we will continue to use dbfs.  Maybe in the future when volumes are supported by scala io we can re-evaluate, but for now dbfs seems the way to go.