cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT Unity catalog schema no storage location mention details

kiko_roy
Contributor

Hi Team

As part of an earlier That I did with Databricks team , I got the info for that if one wants to dump the data in unity catalog schema from a DLT pipeline , The specific schema's Storage location must not be specified . else he DLT pipeline will fail at the "waiting Resources" stage.
Question 1: I want to know if DLT picks up a location by default in Unity catalog where is this location. Does it store the data files in the ADLS location tagged to the meta store? 

Question2: if I have two catalogs(say for business module 1 and business module 2)  tagged to same meta store, and I want to adopt DLT pipeline for business module1, would the users of business module2 have access to data of business module 1. Is there any access concern ?

 

1 ACCEPTED SOLUTION

Accepted Solutions

Walter_C
Honored Contributor
Honored Contributor

Question 1: Delta Live Tables (DLT) picks up a location by default in Unity catalog (UC) for storing the data. This location is in the root storage that is configured when you create the metastore. Managed tables, which are the default way to create tables in UC, are stored in this root storage location. If you have specified managed table storage locations at the catalog or schema levels, these will override the root storage location. As for whether it stores the data files in the ADLS location tagged to the meta store, it depends on the cloud provider and storage configuration.
Question 2: Access to data in different catalogs is governed by permissions. Users must have the USE SCHEMA permissions on the schema, and they must have the USE CATALOG permission on its parent catalog. To query a table, users must have the SELECT permission on the table, the USE SCHEMA permission on its parent schema, and the USE CATALOG permission on its parent catalog. Therefore, if the users of business module 2 do not have the necessary permissions for the catalog or schema of business module 1, they would not have access to its data.

View solution in original post

2 REPLIES 2

Walter_C
Honored Contributor
Honored Contributor

Question 1: Delta Live Tables (DLT) picks up a location by default in Unity catalog (UC) for storing the data. This location is in the root storage that is configured when you create the metastore. Managed tables, which are the default way to create tables in UC, are stored in this root storage location. If you have specified managed table storage locations at the catalog or schema levels, these will override the root storage location. As for whether it stores the data files in the ADLS location tagged to the meta store, it depends on the cloud provider and storage configuration.
Question 2: Access to data in different catalogs is governed by permissions. Users must have the USE SCHEMA permissions on the schema, and they must have the USE CATALOG permission on its parent catalog. To query a table, users must have the SELECT permission on the table, the USE SCHEMA permission on its parent schema, and the USE CATALOG permission on its parent catalog. Therefore, if the users of business module 2 do not have the necessary permissions for the catalog or schema of business module 1, they would not have access to its data.

kiko_roy
Contributor

Thanks @Walter_C  for the explanation and confirming the understanding. Really Appreaciate

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!