DLT can't authenticate with kinesis using instance profile

[israelst]

When running my notebook using personal compute with instance profile I am indeed able to readStream from kinesis. But adding it as a DLT with UC, while specifying the same instance-profile in the DLT pipeline setting - causes a "MissingAuthentication" exception from kinesis...

If I am using the hive_metastore in the DLT - it does work! Why in UC it does not?!

[brockb]

Hi,

Please review the "Limitations" section of this Unity Catalog DLT document. Do one or more of these described situations applicable to you, such as possibly:

Existing pipelines that use the Hive metastore cannot be upgraded to use Unity Catalog. To migrate an existing pipeline that writes to Hive metastore, you must create a new pipeline and re-ingest data from the data source(s).

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#limitations

Thanks.

[Mathias_Peters]

Hi, were you able to solve this problem? If so, what was the solution?

[Babu_Krishnan]

Hello, Did you fix this problem ? We are having similar issue with SQS permission.