topic Re: AWS Instance Profiles and DLT Pipelines in Data Engineering

AWS Instance Profiles and DLT Pipelines

youngchef — Mon, 01 Aug 2022 23:27:03 GMT

Hey everyone!

I'm building a DLT pipeline that reads files from S3 (or tries to) and then writes them into different directories in my s3 bucket. The problem is I usually access S3 with an instance profile attached to a cluster, but DLT does not give me the option to use an instance profile for the job cluster it creates.

What is the solution here? Do I somehow have to pass my AWS keys in the DLT notebook?

Re: AWS Instance Profiles and DLT Pipelines

Prabakar — Tue, 02 Aug 2022 10:53:58 GMT

hi @Quinn Harty If you need an instance profile or other configuration to access your storage location, specify it for both the default cluster and the maintenance cluster.

https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-configuration.html#cluster-configuration

Re: AWS Instance Profiles and DLT Pipelines

Hubert-Dudek — Tue, 02 Aug 2022 13:49:52 GMT

{
  "clusters": [
    {
      "label": "default",
      "aws_attributes": {
        "instance_profile_arn": "arn:aws:..."
      }
    },
    {
      "label": "maintenance",
      "aws_attributes": {
        "instance_profile_arn": "arn:aws:..."
      }
    }
  ]
}

Re: AWS Instance Profiles and DLT Pipelines

jose_gonzalez — Wed, 17 Aug 2022 22:15:14 GMT

Hi @Quinn Harty,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.