Python module import with Dedicated access mode

[FedeRaimondi]

I currently have a repo connected in databricks and I was able to correctly import a python module from src folder located in the same root.

Since I am using a Machine Learning runtime, I am force to choose a Dedicated (formerly: Single user) access mode. The module import works when I use directly my user, but if I change this into a group (to which I'm part of) then "src" module can't be find.

The weird thing is that when I run sys.path, I get the same exact list of paths.

So I am probably missing some knowledge of how the access works when using a ML runtime and assigning a group instead of a single user. Is there a way to import correctly the module?

[szymon_dybczak · Accepted solution]

Hi @FedeRaimondi ,

It could be permission issue. According to documentation, when a compute resource has Dedicated access, the resource can be assigned to a single user or a group. When assigned to a group (a group cluster), the user's permissions automatically down-scopes to the group's permissions, allowing the user to securely share the resource with other members of the group.
So maybe when you setup repo as a user the group has no access to that because of this down-scoped behaviour.

Databricks Runtime for Machine Learning | Databricks Documentation

I think this could be the case:

"For example, the group needs explicit permission to query a table, access a secret scope or secret, use a Unity Catalog connection credential, access a Git folder, or create a workspace object."

Assign compute resources to a group | Databricks Documentation

And you can check also best practices for managing group clusters:

Assign compute resources to a group | Databricks Documentation

[FedeRaimondi]

Thanks @szymon_dybczak ! I confirm that's a permission issue and assigning "CAN MANAGE" solves it.

I still find it not really intuitive, since the goal is to use a shared cluster (with ML runtime) for development purposes. I mean, it would make sense when following the recommended approach of storing your code in "/Workspace/Groups/<groupName>", but for the integration with Git Folders it really adds some complexity.

The whole team assigned to the group will need to grant access to their Git Folders 😅

[szymon_dybczak]

Hi @FedeRaimondi ,

Glad that it worked for you. And I agree with you that it's bit confusing 🙂