topic Re: DLT can't authenticate with kinesis using instance profile in Data Engineering

DLT can't authenticate with kinesis using instance profile

israelst — Mon, 15 Jan 2024 08:57:56 GMT

When running my notebook using personal compute with instance profile I am indeed able to readStream from kinesis. But adding it as a DLT with UC, while specifying the same instance-profile in the DLT pipeline setting - causes a "MissingAuthentication" exception from kinesis...

If I am using the hive_metastore in the DLT - it does work! Why in UC it does not?!

Re: DLT can't authenticate with kinesis using instance profile

brockb — Tue, 06 Feb 2024 03:06:09 GMT

Hi,

Please review the "Limitations" section of this Unity Catalog DLT document. Do one or more of these described situations applicable to you, such as possibly:

Existing pipelines that use the Hive metastore cannot be upgraded to use Unity Catalog. To migrate an existing pipeline that writes to Hive metastore, you must create a new pipeline and re-ingest data from the data source(s).

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#limitations

Thanks.

Re: DLT can't authenticate with kinesis using instance profile

Mathias_Peters — Thu, 25 Apr 2024 13:25:29 GMT

Hi, were you able to solve this problem? If so, what was the solution?

Re: DLT can't authenticate with kinesis using instance profile

Babu_Krishnan — Thu, 02 May 2024 17:28:09 GMT

Hello, Did you fix this problem ? We are having similar issue with SQS permission.

Re: DLT can't authenticate with kinesis using instance profile

Mathias_Peters — Tue, 07 May 2024 12:57:31 GMT

We switched to the preview channel and used a roleArn as param. That worked with DLT.

Re: DLT can't authenticate with kinesis using instance profile

Babu_Krishnan — Tue, 07 May 2024 13:43:41 GMT

@Mathias_Peters , Thanks for the details. Curious how make the roleAan part work , we are able to make it work only with passing accessKey and Secret key, not with roleArn. if you are using SQL based DLT tables , Could you please share some code samples on how you are passing the roleArn info?

Re: DLT can't authenticate with kinesis using instance profile

Mathias_Peters — Wed, 08 May 2024 10:53:51 GMT

We have used the roleArn and role session name like this:

CREATE STREAMING TABLE table_name as SELECT * FROM STREAM read_kinesis ( streamName => 'stream', initialPosition => 'earliest', roleArn => 'arn:aws:iam::ACCT_ID:role/ROLE_NAME', roleSessionName => 'databricks' );

The service principal executing the pipeline has to be able to assume the role referenced by roleArn.

Re: DLT can't authenticate with kinesis using instance profile

Babu_Krishnan — Wed, 08 May 2024 13:58:12 GMT

Thanks for sharing the details @Mathias_Peters , Let us try this.

Re: DLT can't authenticate with kinesis using instance profile

am1go — Sat, 03 May 2025 17:26:07 GMT

Has anyone figured it out? Tried all solutions posted in this thread, nothing works for me...