DLT can't authenticate with kinesis using instance profile

[israelst]

When running my notebook using personal compute with instance profile I am indeed able to readStream from kinesis. But adding it as a DLT with UC, while specifying the same instance-profile in the DLT pipeline setting - causes a "MissingAuthentication" exception from kinesis...

If I am using the hive_metastore in the DLT - it does work! Why in UC it does not?!

[brockb]

Hi,

Please review the "Limitations" section of this Unity Catalog DLT document. Do one or more of these described situations applicable to you, such as possibly:

Existing pipelines that use the Hive metastore cannot be upgraded to use Unity Catalog. To migrate an existing pipeline that writes to Hive metastore, you must create a new pipeline and re-ingest data from the data source(s).

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#limitations

Thanks.

[Mathias_Peters]

Hi, were you able to solve this problem? If so, what was the solution?

[Babu_Krishnan]

Hello, Did you fix this problem ? We are having similar issue with SQS permission.

[Mathias_Peters]

We switched to the preview channel and used a roleArn as param. That worked with DLT.

[Babu_Krishnan]

@Mathias_Peters , Thanks for the details. Curious how make the roleAan part work , we are able to make it work only with passing accessKey and Secret key, not with roleArn. if you are using SQL based DLT tables , Could you please share some code samples on how you are passing the roleArn info?

[Mathias_Peters]

We have used the roleArn and role session name like this:

CREATE STREAMING TABLE table_name
 as SELECT * FROM STREAM read_kinesis (
        streamName => 'stream',
        initialPosition => 'earliest',
        roleArn => 'arn:aws:iam::ACCT_ID:role/ROLE_NAME',
        roleSessionName => 'databricks'
        );

The service principal executing the pipeline has to be able to assume the role referenced by roleArn. 

[Babu_Krishnan]

Thanks for sharing the details @Mathias_Peters  , Let us try this.