- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2022 11:16 PM
You don't need to mount to dbfs. The security issue with mounting is that as soon as you've mounted using a credential, everyone with access to the workspace now has access to the data at that mounted location.
The recommended way that isn't mounting to dbfs is to use session scoped connections using provider secret scopes (azure key vault, aws parameter store, etc) and access control lists. This way, you have a service principal/iam role that has access to the storage location, and you control who has access to the secrets for that service principal. I personally put all of my databricks artifacts in repos under the following folder setup:
databricks>notebooks>category>artifact
databricks>functions>category.py
If you were using azure, you could have a function called set_session_scope in databricks.functions.azure and then you could just import the function and pass it the required parameters using from databricks.functions.azure import set_session_scope.