I encountered an issue where I couldn’t successfully connect to Kinesis Data Streams using instance profile authentication while working with Delta Live Tables (DLT) in a Unity Catalog (UC)-enabled environment.

According to the documentation, instance profiles are not supported in shared access mode. On the other hand, UC-enabled pipelines must run in shared access mode.

https://docs.databricks.com/en/connect/streaming/kinesis.html#authenticate-with-amazon-kinesis

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#requirements

If alternative authentication methods are not an option (e.g., due to organizational security policies prohibiting the issuance of AWS access keys), my understanding is that UC-enabled DLT cannot be used in this scenario.

In contrast, I have confirmed that using Hive Metastore allows a successful connection to Kinesis with instance profile authentication.

If anyone has ideas or workarounds for this limitation, please share them here.