topic Re: best way to store config files in a Unity workspace (Scala/typesafe) in Data Engineering

best way to store config files in a Unity workspace (Scala/typesafe)

-werners- — Fri, 20 Oct 2023 11:20:20 GMT

We use typesafe (scala) to read configuration values from hocon files.
When not using Unity, we read the configuration files from /dbfs/... works fine.
However, with Unity, usage of dbfs is frowned upon.
So I started looking into alternatives.
And unfortunately, dbfs still seems the best and easiest solution.
Anyone that has a better option which I may have missed?

And no, porting all code to python is not an option 🙂

Re: best way to store config files in a Unity workspace (Scala/typesafe)

-werners- — Fri, 20 Oct 2023 12:58:04 GMT

Hi Kaniz, thanks for the suggestion.
However in our case storing config values in env vars will be cumbersome. We put all of our filters applied in notebooks in those config files, together with default selections for entities etc.
So they are not 'sensitive' values.
I tried using volumes, but unfortunately volumes do not (yet) support scala.io (although it would totally make sense to use volumes here).

Re: best way to store config files in a Unity workspace (Scala/typesafe)

-werners- — Mon, 18 Dec 2023 10:43:05 GMT

In the end we will continue to use dbfs. Maybe in the future when volumes are supported by scala io we can re-evaluate, but for now dbfs seems the way to go.