cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Cant Use Delta Live Tables to read MSK using IAM authenthication

OluPopoola
New Contributor II

Hi All

I am trying to use Delta Live Tables to connect to MSK.

We have set up serverless MSK clusters that use IAM for its authetication. I cannot connect to it from a dlt notebook. The same code near enough works on normal clusters that have java libraries for IAM installed. The data is fetched from MSK and written to delta tables.  Porting the same code to work on DLT is proving impossible because of the python or SQL constraints. Has anyone done this successfully? Can I see the snippet on the connection to kafka?

mine is this:

kafka_options = {
    "kafka.bootstrap.servers": kafka_bootstrap_servers,
    "subscribe": ",".join(topic_to_table.keys()),
    "kafka.security.protocol": "SASL_SSL",
    "kafka.sasl.mechanism": "AWS_MSK_IAM",
    "startingOffsets": "earliest",
    "kafka.ssl.endpoint.identification.algorithm": "",
    # "kafka.sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
}
The error is this: Timed out waiting for a node assignment.
1 REPLY 1

OluPopoola
New Contributor II

Just rephrasing the question:

I am trying to use the DLT to connect to serverless MSK clusters authenticated by IAM. The code works on ordinary clusters but doesn't work when run on DLT clusters. I think the issue is the authentication because we can use java files on DLT clusters. My questions are these:
 
Am I wrong in saying we can't install maven libraries on DLT clusters, If yes how can we do this?
 
Is there an example (code snippet)  of the kafka options settings that use IAM for authentication.
I am using this and the code is timing out
kafka_options = {
    "kafka.bootstrap.servers": kafka_bootstrap_servers,
    "subscribe": ",".join(topic_to_table.keys()),
    "kafka.security.protocol": "SASL_SSL",
    "kafka.sasl.mechanism": "AWS_MSK_IAM",
    "startingOffsets": "earliest",
    "kafka.ssl.endpoint.identification.algorithm": ""
    # "kafka.sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
}
If I leave in the line I took out in, it fails the code fails because it cant create a kafkaClient. If I change it to
"shadedmskiam.software.amazon.msk.auth.iam.IAMLoginModule required;" It complains it cant find the class
Help.
 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group