cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

best way to store config files in a Unity workspace (Scala/typesafe)

-werners-
Esteemed Contributor III

We use typesafe (scala) to read configuration values from hocon files.
When not using Unity, we read the configuration files from /dbfs/...  works fine.
However, with Unity, usage of dbfs is frowned upon.
So I started looking into alternatives.
And unfortunately, dbfs still seems the best and easiest solution.
Anyone that has a better option which I may have missed?

And no, porting all code to python is not an option ๐Ÿ™‚

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

In the end we will continue to use dbfs.  Maybe in the future when volumes are supported by scala io we can re-evaluate, but for now dbfs seems the way to go.

View solution in original post

2 REPLIES 2

-werners-
Esteemed Contributor III

Hi Kaniz, thanks for the suggestion.
However in our case storing config values in env vars will be cumbersome.  We put all of our filters applied in notebooks in those config files, together with default selections for entities etc.
So they are not 'sensitive' values.
I tried using volumes, but unfortunately volumes do not (yet) support scala.io (although it would totally make sense to use volumes here).

-werners-
Esteemed Contributor III

In the end we will continue to use dbfs.  Maybe in the future when volumes are supported by scala io we can re-evaluate, but for now dbfs seems the way to go.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group