- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2022 08:25 PM
Hi Team,
Can anyone please help me figure out how to configure Delta Live Tables Cluster accessing AWS S3 bucket without Instance profile defined in Cluster's JSON?
The idea is, the user who is running the DLT Cluster has Storage Credentials and External Locations fully granted so that the user can read/write files in the S3 bucket from Notebook through compute cluster without Instance profile attached on it
When the user is running DLT cluster, however, the cluster runs into access denied error without instance profile attached in its JSON configuration.
My assumption was, the DLT cluster was supposed to act exactly same as Compute cluster@Notebook, but DLT cluster ran into an error.
Any comment or guidance would be very much appreciated
Thanks in advance
Cheers,
J
- Labels:
-
Instance Profile
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-05-2022 04:35 AM
Hi @Jason Nam , DLT and unity catalog are not integrated yet. The cluster-notebook setup uses UC and can access S3 but not the DLT jobs. Please check the limitations in this document (7th point):
https://docs.databricks.com/release-notes/unity-catalog/20220825.html#unity-catalog-ga-limitations
We can expect this integration in the upcoming quarters.
Hope this information helps.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-07-2022 01:43 AM
Thank you so much for your reply and confirmation.
It helps me understand DLT + UC further
Cheers,
