topic Re: DLT Cluster accessing to S3 bucket without Instance Profile attached in Data Engineering

DLT Cluster accessing to S3 bucket without Instance Profile attached

JasonN — Mon, 05 Sep 2022 03:25:40 GMT

Hi Team,

Can anyone please help me figure out how to configure Delta Live Tables Cluster accessing AWS S3 bucket without Instance profile defined in Cluster's JSON?

The idea is, the user who is running the DLT Cluster has Storage Credentials and External Locations fully granted so that the user can read/write files in the S3 bucket from Notebook through compute cluster without Instance profile attached on it

When the user is running DLT cluster, however, the cluster runs into access denied error without instance profile attached in its JSON configuration.

My assumption was, the DLT cluster was supposed to act exactly same as Compute cluster@Notebook, but DLT cluster ran into an error.

Any comment or guidance would be very much appreciated

Thanks in advance

Cheers,

Re: DLT Cluster accessing to S3 bucket without Instance Profile attached

Vivian_Wilfred — Mon, 05 Sep 2022 11:35:54 GMT

Hi @Jason Nam , DLT and unity catalog are not integrated yet. The cluster-notebook setup uses UC and can access S3 but not the DLT jobs. Please check the limitations in this document (7th point):

https://docs.databricks.com/release-notes/unity-catalog/20220825.html#unity-catalog-ga-limitations

We can expect this integration in the upcoming quarters.

Hope this information helps.

Re: DLT Cluster accessing to S3 bucket without Instance Profile attached

JasonN — Wed, 07 Sep 2022 08:43:28 GMT

Thank you so much for your reply and confirmation.

It helps me understand DLT + UC further

Cheers,