Limitations When Using Instance Profiles to Connect to Kinesis

[Takuya-Omi]

I encountered an issue where I couldn’t successfully connect to Kinesis Data Streams using instance profile authentication while working with Delta Live Tables (DLT) in a Unity Catalog (UC)-enabled environment.

According to the documentation, instance profiles are not supported in shared access mode. On the other hand, UC-enabled pipelines must run in shared access mode.

https://docs.databricks.com/en/connect/streaming/kinesis.html#authenticate-with-amazon-kinesis

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#requirements

If alternative authentication methods are not an option (e.g., due to organizational security policies prohibiting the issuance of AWS access keys), my understanding is that UC-enabled DLT cannot be used in this scenario.

In contrast, I have confirmed that using Hive Metastore allows a successful connection to Kinesis with instance profile authentication.

I’m sharing this because it’s a recent issue that I found a bit challenging.

If anyone has ideas or workarounds for this limitation, please share them here.

[Alberto_Umana]

Hi @Takuya-Omi,

Looks like this is a workaround: https://community.databricks.com/t5/data-engineering/dlt-can-t-authenticate-with-kinesis-using-insta...

[Takuya-Omi]

@Alberto_Umana 

Thank you for sharing. However, I have already followed the steps mentioned in the article, and I’m still unable to establish a connection.

When using AWS access keys, the connection is successful, which confirms that there are no issues with access to Kinesis or the network configuration.
Additionally, the connection works with DLT pipelines that are not UC-enabled, so it seems unlikely that there are any errors in the IAM roles or policies configured for the instance profile.