Just rephrasing the question:
I am trying to use the DLT to connect to serverless MSK clusters authenticated by IAM. The code works on ordinary clusters but doesn't work when run on DLT clusters. I think the issue is the authentication because we can use java files on DLT clusters. My questions are these:
Am I wrong in saying we can't install maven libraries on DLT clusters, If yes how can we do this?
Is there an example (code snippet) of the kafka options settings that use IAM for authentication.
I am using this and the code is timing out
kafka_options = {
"kafka.bootstrap.servers": kafka_bootstrap_servers,
"subscribe": ",".join(topic_to_table.keys()),
"kafka.security.protocol": "SASL_SSL",
"kafka.sasl.mechanism": "AWS_MSK_IAM",
"startingOffsets": "earliest",
"kafka.ssl.endpoint.identification.algorithm": ""
# "kafka.sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
}
If I leave in the line I took out in, it fails the code fails because it cant create a kafkaClient. If I change it to
"shadedmskiam.software.amazon.msk.auth.iam.IAMLoginModule required;" It complains it cant find the class
Help.