topic Re: Limitations When Using Instance Profiles to Connect to Kinesis in Data Engineering

Limitations When Using Instance Profiles to Connect to Kinesis

Takuya-Omi — Mon, 13 Jan 2025 14:16:55 GMT

I encountered an issue where I couldn’t successfully connect to Kinesis Data Streams using instance profile authentication while working with Delta Live Tables (DLT) in a Unity Catalog (UC)-enabled environment.

According to the documentation, instance profiles are not supported in shared access mode. On the other hand, UC-enabled pipelines must run in shared access mode.

https://docs.databricks.com/en/connect/streaming/kinesis.html#authenticate-with-amazon-kinesis

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html#requirements

If alternative authentication methods are not an option (e.g., due to organizational security policies prohibiting the issuance of AWS access keys), my understanding is that UC-enabled DLT cannot be used in this scenario.

In contrast, I have confirmed that using Hive Metastore allows a successful connection to Kinesis with instance profile authentication.

I’m sharing this because it’s a recent issue that I found a bit challenging.

If anyone has ideas or workarounds for this limitation, please share them here.

Re: Limitations When Using Instance Profiles to Connect to Kinesis

Alberto_Umana — Mon, 13 Jan 2025 14:19:59 GMT

Hi @Takuya-Omi,

Looks like this is a workaround: https://community.databricks.com/t5/data-engineering/dlt-can-t-authenticate-with-kinesis-using-instance-profile/m-p/68570#M33702

Re: Limitations When Using Instance Profiles to Connect to Kinesis

Takuya-Omi — Mon, 13 Jan 2025 15:43:36 GMT

@Alberto_Umana

Thank you for sharing. However, I have already followed the steps mentioned in the article, and I’m still unable to establish a connection.

When using AWS access keys, the connection is successful, which confirms that there are no issues with access to Kinesis or the network configuration.
Additionally, the connection works with DLT pipelines that are not UC-enabled, so it seems unlikely that there are any errors in the IAM roles or policies configured for the instance profile.

Re: Limitations When Using Instance Profiles to Connect to Kinesis

am1go — Sat, 03 May 2025 22:59:08 GMT

I'm in the same boat - tried every workaround possible, nothing works for me. Databricks is pushing Unity Catalog hard so i find it unsettling that there is no solution for this issue other than reverting back to using hive metastore.