Store a secret only accessible to the current user

dvmentalmadess — Tue, 12 Apr 2022 16:13:56 GMT

During an interactive notebook session, I want a user to be able to retrieve a secret specific to that user. I haven't decided on storage mechanisms, but I'm open to storage mechanisms that can scalably authorize access to a single user and that I can write the secret from an external service. I have looked into the following:

Databricks Secrets: with a limit of 100 scopes, this does not scale beyond 100 users and I work in an engineering organization with over 200 people
IAM credential passthrough: does not support MLFlow (my data science team uses MLFlow), and according to my reading it does not support non-admin users calling Scala (I have at least one team that requires the use of Scala)
Table Access Control: I could use this to create a view that is limited to results matching CURRENT_USER, but won't work for users who need to use Scala
Workspace object access control: it has an API I can use to write secrets, and I can limit access by user. I would prefer if I can prevent admins from reading the secret of another user, but I haven't figured out if this is possible yet.

I'm thinking workspace object access control is a good option. Can anyone tell me if admin users automatically have access to all objects in a workspace? Is there anything I may have missed that would compromise this solution? Are any of my assumptions incorrect? Are there viable alternatives I'm missing?

Re: Store a secret only accessible to the current user

dvmentalmadess — Fri, 05 Aug 2022 18:29:47 GMT

I ended up using Databricks Secrets as the storage mechanism after learning from my account rep that the limit is soft and we can request a higher scope limit. In this case, each user gets a dedicated scope and no other users have access.

topic Store a secret only accessible to the current user in Machine Learning

Store a secret only accessible to the current user

Re: Store a secret only accessible to the current user