topic Re: Delta Sharing - Unity Catalog difference in Data Engineering

Delta Sharing - Unity Catalog difference

Anonymous — Sun, 06 Jun 2021 04:50:06 GMT

Delta Sharing and Unity catalog both have elements of data sharing. Can you please explain when one would use Delta sharing vs Unity Catalog?

Re: Delta Sharing - Unity Catalog difference

User16826993440 — Mon, 07 Jun 2021 10:46:00 GMT

Unity Catalog provides the capability to discover, audit and govern data assets in one place. This capability does not have to utilize data sharing as it can help you manage your self managed Hive, S3, Azure Data Lake, or Google Cloud storages. If you wanted to perform this management of data across organizations or functions, this is where Delta Sharing would be utilized in conjunction with Unity Catalog, allowing for data to be shared across organizations through Delta and managed/governed through the Unity Catalog.

Re: Delta Sharing - Unity Catalog difference

prasadvaze — Tue, 23 Nov 2021 05:35:03 GMT

@Semir Redzepagic I use azure and have Dev, Test, Prod subscription. I have multiple databricks workspaces in each subscription. Can the unity catalog in each subscription /environment be separate or the same? ( obviously, i do not want the catalog in dev env to see/access tables in prod environment)

Re: Delta Sharing - Unity Catalog difference

User16826993440 — Wed, 24 Nov 2021 14:52:17 GMT

@prasad vaze UC is managed at the account level by the account admins so you have the ability to manage across all workspaces/environments on the account.

Re: Delta Sharing - Unity Catalog difference

prasadvaze — Wed, 24 Nov 2021 16:07:03 GMT

The unity catalog is not generally available (or at least it's not enabled in my company's databricks account ) so can not try it out. My question is "can I create separate catalogs between dev/test/prod environments ( multiple catalogs under one account) ? I hope yes but can not try this yet. In each environment, I have multiple databricks workspaces and I understand that one catalog in one environment can administer all workspaces in it. The reason for separate catalog per environment (though all environments are under the same databricks account) is separation of duties and security

Re: Delta Sharing - Unity Catalog difference

User16826993440 — Mon, 29 Nov 2021 14:07:33 GMT

Yes - you can control the data available in each of the environments/workspaces. If there are security questions, please reach out to your Databricks account team to further discuss your specific scenario.

Re: Delta Sharing - Unity Catalog difference

DBXC — Fri, 12 May 2023 14:49:10 GMT

Based on the Databricks reply from the post below:

"Unity Catalog does not currently support separating data by workspace or Azure subscription. As you noted, data from all catalogs within a region can be accessed by any workspace within that region, and it is up to user permissions to restrict access appropriately."

https://community.databricks.com/s/question/0D58Y00009kfFo4SAE/how-to-segregate-devstagingprod-data-and-resources-with-unity-catalog