topic Re: Add an Instance Profile to a DLT job cluster in Data Engineering

Add an Instance Profile to a DLT job cluster

BradSheridan — Thu, 14 Jul 2022 16:49:29 GMT

@Tomasz Bacewicz I've got another, related question for you about the job cluster that is spun up for DLT jobs. Adding the JSON strings for our required E2 tags worked like a charm, but now I need to attach an existing Instance Profile since I'm trying to access our S3 buckets in my DLT job. I've looked online for how to do this in the JSON but no luck. thanks! Brad

Re: Add an Instance Profile to a DLT job cluster

tomasz — Thu, 14 Jul 2022 17:04:42 GMT

@Brad Sheridan To do that you have to add the aws_attributes tag within a cluster configuration and there you have the ability to add an instance_profile_arn like so:

"clusters": [

{

"label": "default",

"aws_attributes": {

"instance_profile_arn": "<REDACTED>"

I hope this helps!

Re: Add an Instance Profile to a DLT job cluster

BradSheridan — Thu, 14 Jul 2022 18:13:43 GMT

Once again, THANK YOU! Worked like a charm