cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Connecting to AWS MSK

Sudhir1
New Contributor II

how to connect to the AWS MSK which has I-Am based authentication?

5 REPLIES 5

karthik_p
Esteemed Contributor

@Sudhir Jaiswalโ€‹ please try to check below stackflow link and download new jar file related to msk iam and attach to cluster that you will using and add read config and provide topic name and other details and see

https://stackoverflow.com/questions/69772612/kafka-consumer-using-aws-msk-iam-classcastexception-err...

Sudhir1
New Contributor II

@karthik_p

It seems I made some progress with the approach mentioned. But I get an error:

Caused by: kafkashaded.org.apache.kafka.common.errors.SaslAuthenticationException: [dabd4181-e46b-4d41-8e01-87b4a0ec9f5a]: Access denied

I am wondering what else I am missing!

judyy
New Contributor III

Hi Karthik,

I've followed the this solution, but got the error like this: SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Failed to find AWS IAM Credentials [Caused by aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain@62c66486: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@7881f590: Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), aws_msk_iam_auth_shadow.com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@362f07a1: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]]]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.

But I've already set up the OS environment with below code:

os.environ['AWS_ACCESS_KEY_ID'] = "AKID"

os.environ['AWS_SECRET_ACCESS_KEY'] = "XXXXXXXXXXXX"

Could you give me any insights of it? Many thanks in advance!

judyy
New Contributor III

I fixed this error by adding 2 environment variables under cluster configuration -> advanced options -> Environment variables:

AWS_SECRET_ACCESS_KEY="XXXXXX"

AWS_ACCESS_KEY_ID="XXXX"

AWS_DEFAULT_REGION="XXXX"โ€‹

Anonymous
Not applicable

Hi @Sudhir Jaiswalโ€‹ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group