cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT Cluster accessing to S3 bucket without Instance Profile attached

JasonN
New Contributor II

Hi Team,

Can anyone please help me figure out how to configure Delta Live Tables Cluster accessing AWS S3 bucket without Instance profile defined in Cluster's JSON?

The idea is, the user who is running the DLT Cluster has Storage Credentials and External Locations fully granted so that the user can read/write files in the S3 bucket from Notebook through compute cluster without Instance profile attached on it

When the user is running DLT cluster, however, the cluster runs into access denied error without instance profile attached in its JSON configuration.

My assumption was, the DLT cluster was supposed to act exactly same as Compute cluster@Notebook, but DLT cluster ran into an error.

Any comment or guidance would be very much appreciated

Thanks in advance

Cheers,

J

1 ACCEPTED SOLUTION

Accepted Solutions

Vivian_Wilfred
Honored Contributor

Hi @Jason Nam​ , DLT and unity catalog are not integrated yet. The cluster-notebook setup uses UC and can access S3 but not the DLT jobs. Please check the limitations in this document (7th point):

https://docs.databricks.com/release-notes/unity-catalog/20220825.html#unity-catalog-ga-limitations

We can expect this integration in the upcoming quarters.

Hope this information helps.

View solution in original post

2 REPLIES 2

Vivian_Wilfred
Honored Contributor

Hi @Jason Nam​ , DLT and unity catalog are not integrated yet. The cluster-notebook setup uses UC and can access S3 but not the DLT jobs. Please check the limitations in this document (7th point):

https://docs.databricks.com/release-notes/unity-catalog/20220825.html#unity-catalog-ga-limitations

We can expect this integration in the upcoming quarters.

Hope this information helps.

Thank you so much for your reply and confirmation.

It helps me understand DLT + UC further

Cheers,

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group