topic Re: Connecting AWS MSK from Databricks in Data Engineering

Connecting AWS MSK from Databricks

DB_Five — Wed, 04 Dec 2024 12:09:29 GMT

Hello ,

I am new to AWS and MSK. I have created MSK on VPC with public subnet and I am trying to connect it from Databricks on AWS. I see that both MSK and VPC are in two different VPC.

Do we need to create VPC peering to establish the connection between the two. I have not seen any documentation or any thread mentioning this setup.

Can someone help me with this.

Regards,

Harsha

Re: Connecting AWS MSK from Databricks

Louis_Frolio — Wed, 04 Dec 2024 12:16:50 GMT

Greetings DB_Five, To connect an Amazon Managed Streaming for Apache Kafka (MSK) cluster from a Databricks workspace on AWS when both are in different Virtual Private Clouds (VPCs), you will need to establish a network connection between the two VPCs.

Several options are available:

1. VPC Peering

2. AWS PrivateLink

3. AWS Transit Gateway

Hope this helps. Louis.

Re: Connecting AWS MSK from Databricks

DB_Five — Wed, 04 Dec 2024 13:35:13 GMT

Hello,

Thanks for your quick update.

I will continue with the setup. I also have one more question.

It is mentioned in the set up that, we have to have the bellow in the Kafka client properties.

In addition, if you choose to configure your connection using an IAM role, you must modify the value provided to kafka.sasl.jaas.config to include the role ARN, as in the following example: shadedmskiam.software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn="arn:aws:iam::123456789012:role/msk_client_role"

Do we have to createa new IAM role with permissions to access Kafka or we can use the IAM role associated with the workspace creation.

Regards,
Harsha

Re: Connecting AWS MSK from Databricks

Louis_Frolio — Wed, 04 Dec 2024 13:51:35 GMT

When configuring your connection to Amazon Managed Streaming for Apache Kafka (MSK) using an IAM role, you have the option to use either a new IAM role specifically created for this purpose or the IAM role associated with your Databricks workspace.

Hope this helps. Louis