topic READ FILES and WRITE FILES when using Hive Metastore in Data Governance

READ FILES and WRITE FILES when using Hive Metastore

knawara — Wed, 15 Feb 2023 12:33:38 GMT

Hello,

I'm confused about documentation on privilege types when using HMS.

The following page is supposed to talk about HMS

https://docs.databricks.com/sql/language-manual/sql-ref-privileges-hms.html

but it also mentions

READ FILES

Query files directly using the storage credential or external location.

WRITE FILES

Directly COPY INTO files governed by the storage credential or external location.

If I understand correctly these (Storage Credential and External Location) only apply to Unity Catalog, as per this page:

https://docs.databricks.com/sql/language-manual/sql-ref-external-locations.html

Is this a mistake in a documentation or there is something more fundamental that I don't understand?

Re: READ FILES and WRITE FILES when using Hive Metastore

Lakshay — Wed, 15 Feb 2023 12:56:55 GMT

Hi @Chris Nawara , The Privilege types and Secure objects are available both in HMS and Unity Catalog. However, there is a difference in implementation across both of them. And as the document mentions "The privilege model and securable objects differ depending on whether you are using a Unity Catalog metastore or the legacy Hive metastore"

Re: READ FILES and WRITE FILES when using Hive Metastore

knawara — Wed, 15 Feb 2023 13:11:50 GMT

HI @Lakshay Goel , thanks for the rapid response!

There are two pages in the documentation, one for HMS:

https://docs.databricks.com/sql/language-manual/sql-ref-privileges-hms.html

which claims "This article describes the privilege model for the legacy Hive metastore".,

and one for Unity Catalog:

https://docs.databricks.com/sql/language-manual/sql-ref-privileges.html

This article describes the privilege model for the Unity Catalog.

READ/WRITE FILES are mentioned in both. What I want to clarify is:

Is READ/WRITE FILES a Unity-Catalog-only concept?
If not, what is its meaning when working with HMS?

Re: READ FILES and WRITE FILES when using Hive Metastore

Lakshay — Fri, 17 Feb 2023 11:08:47 GMT

Hi @Chris Nawara , The two documentations talk about data governance. The concept of data governance is not exclusive to Unity Catalog. The difference here is that Unity Catalog helps you in implementing Data Governance at a much more granular level and better than HMS. So, to answer your questions

Is READ/WRITE FILES a Unity-Catalog-only concept? No
If not, what is its meaning when working with HMS? You can read/write a file with both HMS and UC. But how the data governance and security works in two is the difference.

Re: READ FILES and WRITE FILES when using Hive Metastore

knawara — Fri, 17 Feb 2023 16:12:14 GMT

Hi @Lakshay Goel ,

I'm not talking about reading/writing files, but about READ FILES/WRITE FILES permission that can be granted e.g. in the following way:

GRANT READ FILES ON STORAGE CREDENTIAL <storage_credential_name> TO <principal>;

(from https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html)

As you said, that's a governance question and some things are done way better in UC than in HMS (but for certain reasons not dependent on me UC is not an option). But there are differences between the two, so I guess my question is whether I can use this construct with both HMS and UC, or with UC only

Re: READ FILES and WRITE FILES when using Hive Metastore

Anonymous — Tue, 21 Feb 2023 10:16:47 GMT

Hi @Chris Nawara

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: READ FILES and WRITE FILES when using Hive Metastore

knawara — Tue, 21 Feb 2023 11:10:04 GMT

Hi @Vidula Khanna , thanks for checking in! Not yet, my last message is still unanswered

Re: READ FILES and WRITE FILES when using Hive Metastore

Anonymous — Sat, 11 Mar 2023 02:01:14 GMT

Hi @Chris Nawara

I'm sorry you could not find a solution to your problem in the answers provided.

Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.

I suggest providing more information about your problem, such as specific error messages, error logs or details about the steps you have taken. This can help our community members better understand the issue and provide more targeted solutions.

Alternatively, you can consider contacting the support team for your product or service. They may be able to provide additional assistance or escalate the issue to the appropriate section for further investigation.

Thank you for your patience and understanding, and please let us know if there is anything else we can do to assist you.