cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DLT Unity catalog schema no storage location mention details

kiko_roy
Contributor

Hi Team

As part of an earlier That I did with Databricks team , I got the info for that if one wants to dump the data in unity catalog schema from a DLT pipeline , The specific schema's Storage location must not be specified . else he DLT pipeline will fail at the "waiting Resources" stage.
Question 1: I want to know if DLT picks up a location by default in Unity catalog where is this location. Does it store the data files in the ADLS location tagged to the meta store? 

Question2: if I have two catalogs(say for business module 1 and business module 2)  tagged to same meta store, and I want to adopt DLT pipeline for business module1, would the users of business module2 have access to data of business module 1. Is there any access concern ?

 

1 ACCEPTED SOLUTION

Accepted Solutions

Walter_C
Databricks Employee
Databricks Employee

Question 1: Delta Live Tables (DLT) picks up a location by default in Unity catalog (UC) for storing the data. This location is in the root storage that is configured when you create the metastore. Managed tables, which are the default way to create tables in UC, are stored in this root storage location. If you have specified managed table storage locations at the catalog or schema levels, these will override the root storage location. As for whether it stores the data files in the ADLS location tagged to the meta store, it depends on the cloud provider and storage configuration.
Question 2: Access to data in different catalogs is governed by permissions. Users must have the USE SCHEMA permissions on the schema, and they must have the USE CATALOG permission on its parent catalog. To query a table, users must have the SELECT permission on the table, the USE SCHEMA permission on its parent schema, and the USE CATALOG permission on its parent catalog. Therefore, if the users of business module 2 do not have the necessary permissions for the catalog or schema of business module 1, they would not have access to its data.

View solution in original post

2 REPLIES 2

Walter_C
Databricks Employee
Databricks Employee

Question 1: Delta Live Tables (DLT) picks up a location by default in Unity catalog (UC) for storing the data. This location is in the root storage that is configured when you create the metastore. Managed tables, which are the default way to create tables in UC, are stored in this root storage location. If you have specified managed table storage locations at the catalog or schema levels, these will override the root storage location. As for whether it stores the data files in the ADLS location tagged to the meta store, it depends on the cloud provider and storage configuration.
Question 2: Access to data in different catalogs is governed by permissions. Users must have the USE SCHEMA permissions on the schema, and they must have the USE CATALOG permission on its parent catalog. To query a table, users must have the SELECT permission on the table, the USE SCHEMA permission on its parent schema, and the USE CATALOG permission on its parent catalog. Therefore, if the users of business module 2 do not have the necessary permissions for the catalog or schema of business module 1, they would not have access to its data.

kiko_roy
Contributor

Thanks @Walter_C  for the explanation and confirming the understanding. Really Appreaciate

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group