Cant Use Delta Live Tables to read MSK using IAM authenthication

[OluPopoola]

Hi All

I am trying to use Delta Live Tables to connect to MSK.

We have set up serverless MSK clusters that use IAM for its authetication. I cannot connect to it from a dlt notebook. The same code near enough works on normal clusters that have java libraries for IAM installed. The data is fetched from MSK and written to delta tables.  Porting the same code to work on DLT is proving impossible because of the python or SQL constraints. Has anyone done this successfully? Can I see the snippet on the connection to kafka?

mine is this:

kafka_options = {

    "kafka.bootstrap.servers": kafka_bootstrap_servers,

    "subscribe": ",".join(topic_to_table.keys()),

    "kafka.security.protocol": "SASL_SSL",

    "kafka.sasl.mechanism": "AWS_MSK_IAM",

    "startingOffsets": "earliest",

    "kafka.ssl.endpoint.identification.algorithm": "",

    # "kafka.sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"

}

The error is this: Timed out waiting for a node assignment.

[OluPopoola]

Just rephrasing the question:

I am trying to use the DLT to connect to serverless MSK clusters authenticated by IAM. The code works on ordinary clusters but doesn't work when run on DLT clusters. I think the issue is the authentication because we can use java files on DLT clusters. My questions are these:

Am I wrong in saying we can't install maven libraries on DLT clusters, If yes how can we do this?

Is there an example (code snippet)  of the kafka options settings that use IAM for authentication.

I am using this and the code is timing out

kafka_options = {

    "kafka.bootstrap.servers": kafka_bootstrap_servers,

    "subscribe": ",".join(topic_to_table.keys()),

    "kafka.security.protocol": "SASL_SSL",

    "kafka.sasl.mechanism": "AWS_MSK_IAM",

    "startingOffsets": "earliest",

    "kafka.ssl.endpoint.identification.algorithm": ""

    # "kafka.sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"

}

If I leave in the line I took out in, it fails the code fails because it cant create a kafkaClient. If I change it to

"shadedmskiam.software.amazon.msk.auth.iam.IAMLoginModule required;" It complains it cant find the class

Help.