topic Re: Can Unity catalog grant the access to a file inside azure datalake storage? in Data Engineering

Can Unity catalog grant the access to a file inside azure datalake storage?

maaaxx — Tue, 17 Jan 2023 14:42:45 GMT

Hi databricks community,

I have searched quite a while through the internet but did not find an answer. If I have configured the azure datalake connection in Unity data catalog, is it possible to grant the access to users for a specific file or a folder to them? Have seen quite a lot of examples so far for the structured data only.

Thanks.

Re: Can Unity catalog grant the access to a file inside azure datalake storage?

-werners- — Tue, 17 Jan 2023 15:20:56 GMT

No.

Unity catalog enforces permissions on the table level (and catalog and schema etc), but not on the storage level.

Unity itself uses a managed identity or service principal for storage access btw. This id should have access to the data lake.

What you can do is create dynamic views to make a row-level security setup.

https://learn.microsoft.com/en-us/azure/databricks/data-governance/table-acls/object-privileges#row-level-permissions

Re: Can Unity catalog grant the access to a file inside azure datalake storage?

Hubert-Dudek — Tue, 17 Jan 2023 20:31:18 GMT

As @werners said service principal needs to have access to the file level.

In the unity catalog, you can use "READ FILES"/"WRITE FILES" permission to give someone the possibility of reading files from the storage level (but through databricks).

Re: Can Unity catalog grant the access to a file inside azure datalake storage?

maaaxx — Thu, 26 Jan 2023 11:26:04 GMT

Hi @Hubert Dudek @Werner Stinckens , thank you for the idea. In our scenario, we would need to share the files inside the azure datalake in the same folder.

Imagine that we have a folder ORDER001 and file1, file2 and file3. Can we use databricks to share the access to user A the access of file1 and file2 but for user B the access to file3?

Some people have suggested to copy the files outside and create separate container. However, this will unavoidably create duplication and we would like to avoid.

Have you an idea how the acsess control in this scenario could be achieve through databricks?

Many thanks

Re: Can Unity catalog grant the access to a file inside azure datalake storage?

Hubert-Dudek — Thu, 26 Jan 2023 17:08:05 GMT

It is messy as:

Files are in the same folder (so it complicates using an external location and read-write permission)
Unity Catalog is designed to have tables, and you grant access to tables

I don't know what the files are. Unstructured data can be included in the delta file / metastore table (array or binary).

You could also put these files outside of databricks and manage access separately.

@Werner Stinckens, is it possible to have Unity Catalog and mount another storage container under the dbfs path using credentials passthrough?

Re: Can Unity catalog grant the access to a file inside azure datalake storage?

-werners- — Fri, 27 Jan 2023 09:05:56 GMT

I am not sure. Someone at Databricks once told me that mounts and Unity are not friends.

The easiest way to achieve this on file level is either:

not using Unity and use AAD credential passtrough. then define the file access with ACLs on the data lake.
forget about the file access and use dynamic views f.e. to create row level security.

Frankly using ACLs always gets on my nerves. Hard to maintain.