Limitations When Using Instance Profiles to Connect to Kinesis
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 06:16 AM
I encountered an issue where I couldn’t successfully connect to Kinesis Data Streams using instance profile authentication while working with Delta Live Tables (DLT) in a Unity Catalog (UC)-enabled environment.
According to the documentation, instance profiles are not supported in shared access mode. On the other hand, UC-enabled pipelines must run in shared access mode.
https://docs.databricks.com/en/connect/streaming/kinesis.html#authenticate-with-amazon-kinesis
https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#requirements
If alternative authentication methods are not an option (e.g., due to organizational security policies prohibiting the issuance of AWS access keys), my understanding is that UC-enabled DLT cannot be used in this scenario.
In contrast, I have confirmed that using Hive Metastore allows a successful connection to Kinesis with instance profile authentication.
I’m sharing this because it’s a recent issue that I found a bit challenging.
If anyone has ideas or workarounds for this limitation, please share them here.
Takuya Omi (尾美拓哉)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 06:19 AM
Hi @Takuya-Omi,
Looks like this is a workaround: https://community.databricks.com/t5/data-engineering/dlt-can-t-authenticate-with-kinesis-using-insta...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 07:43 AM
Thank you for sharing. However, I have already followed the steps mentioned in the article, and I’m still unable to establish a connection.
When using AWS access keys, the connection is successful, which confirms that there are no issues with access to Kinesis or the network configuration.
Additionally, the connection works with DLT pipelines that are not UC-enabled, so it seems unlikely that there are any errors in the IAM roles or policies configured for the instance profile.
Takuya Omi (尾美拓哉)

