topic Re: Shared cluster configuration that permits `dbutils.fs` commands in Data Engineering

Shared cluster configuration that permits `dbutils.fs` commands

Spencer_Kent — Wed, 07 Jun 2023 23:57:19 GMT

My workspace has a couple different types of clusters, and I'm having issues using the `dbutils` filesystem utilities when connected to a shared cluster. I'm hoping you can help me fix the configuration of the shared cluster so that I can actually use the dbutils filesystem commands.

The workspace is set up to use Unity Catalog, and I'm not sure if that has anything to do with the error.

When I try to `ls` the DBFS root location I get an "INSUFFICIENT_PERMISSIONS" Spark security exception:

The cluster this happens on is a Shared cluster with the data security mode set to "USER_ISOLATION" (by Terraform). It says Unrestricted in the screen shot below, but we set the data security mode in Terraform.

This error does not occur on a Single User cluster with the Individual use policy:

Can you give me guidance on how to configure the shared cluster so that `dbutils.fs.ls("/")` won't error with insufficient permissions?

Thanks you so much!

Re: Shared cluster configuration that permits `dbutils.fs` commands

Anonymous — Fri, 16 Jun 2023 07:08:42 GMT

Hi @Spencer Kent

We haven't heard from you since the last response from @Kaniz Fatma , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Re: Shared cluster configuration that permits `dbutils.fs` commands

Spencer_Kent — Fri, 16 Jun 2023 23:48:44 GMT

Unfortunately the suggestion was not helpful—it is no mystery what the error is (insufficient permissions to access the DBFS root location). What remains a mystery, and the point of my question, is whether there is a certain configuration of shared clusters that is required in order to make the DBFS root location accessible. I ask this question here in this forum because I have not yet found in the Databricks documentation a discussion of this question, and because the terraform provider I'm using does not list any cluster configuration options which seem relevant to making sure the DBFS root location (or any DBFS filesystem utils) are available on a shared cluster.

Re: Shared cluster configuration that permits `dbutils.fs` commands

User16623639898 — Fri, 22 Dec 2023 10:57:32 GMT

Hi @Spencer_Kent ,
Please go through this , https://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog

Shared access mode combines Unity Catalog data governance with Azure Databricks legacy table ACLs. Access to data in the hive_metastore is only available to users that have permissions explicitly granted.

To interact with files directly using DBFS, you must have ANY FILE permissions granted. Because ANY FILE allows users to bypass legacy tables ACLs in the hive_metastore and access all data managed by DBFS, Databricks recommends caution when granting this privilege.

Clusters configured with Single User access mode have full access to DBFS, including all files in the DBFS root and mounted data. DBFS root and mounts are available in this access mode, making it the choice for ML workloads that need access to Unity Catalog datasets.

Databricks recommends using service principals with scheduled jobs and Single User access mode for production workloads that need access to data managed by both DBFS and Unity Catalog.

Re: Shared cluster configuration that permits `dbutils.fs` commands

yosinv — Wed, 17 Jan 2024 10:13:10 GMT

Hello,

I'm encountering a similar issue. We have a team of researchers utilizing a shared cluster without access to the Hive Metastore. I've looked through the documentation, but there doesn't seem to be a way to define or grant "ANY_FILE" during the cluster initialization process.
Moreover, what if Im looking to access the s3 bucket path itself , what is the approach to define it?
please advise

Re: Shared cluster configuration that permits `dbutils.fs` commands

drii_cavalcanti — Wed, 24 Jan 2024 01:15:26 GMT

I could not find `ANY FILE` permission as well.

Re: Shared cluster configuration that permits `dbutils.fs` commands

Nikhil_G — Wed, 31 Jan 2024 10:15:03 GMT

There are two ways to grant access to DBFS using ANY FILE:

To User: GRANT SELECT ON ANY FILE TO '<user_mail_id>'
To Group: GRANT SELECT ON ANY FILE TO '<group_name>'"

Re: Shared cluster configuration that permits `dbutils.fs` commands

ec10826 — Mon, 01 Jul 2024 20:40:30 GMT

I have the same problem that I am receiving "No such file or directory" when trying to access a file in DBFS using a cluster in shared access mode. I am using shared access mode because I want to use table access controls.

I ran "GRANT ALL PRIVILEGES ON ANY FILE TO <user>" and "SHOW GRANTS ON ANY FILE" to verify the grant went through, but am still unable to access the file in DBFS.

@Retired_mod could you advise? Thanks!

Re: Shared cluster configuration that permits `dbutils.fs` commands

jacovangelder — Tue, 02 Jul 2024 06:02:50 GMT

You can't access files on DBFS mounts using a Shared cluster. Either use a Unity Catalog Volume or use a Single user cluster.

Re: Shared cluster configuration that permits `dbutils.fs` commands

ec10826 — Tue, 02 Jul 2024 15:11:20 GMT

@jacovangelder We are using Azure Gov so don't have access to Unity Catalog. What would you suggest we do to control certain users' access to data, since you can't use table access controls with Single user clusters?

Re: Shared cluster configuration that permits `dbutils.fs` commands

jacovangelder — Wed, 03 Jul 2024 05:44:45 GMT

Can you not use a No Isolation Shared cluster with Table access controls enabled on workspace level?