Cant Use Delta Live Tables to read MSK using IAM authenthication
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-08-2024 11:13 AM
Hi All
I am trying to use Delta Live Tables to connect to MSK.
We have set up serverless MSK clusters that use IAM for its authetication. I cannot connect to it from a dlt notebook. The same code near enough works on normal clusters that have java libraries for IAM installed. The data is fetched from MSK and written to delta tables. Porting the same code to work on DLT is proving impossible because of the python or SQL constraints. Has anyone done this successfully? Can I see the snippet on the connection to kafka?
mine is this:
kafka_options = {
"kafka.bootstrap.servers": kafka_bootstrap_servers,
"subscribe": ",".join(topic_to_table.keys()),
"kafka.security.protocol": "SASL_SSL",
"kafka.sasl.mechanism": "AWS_MSK_IAM",
"startingOffsets": "earliest",
"kafka.ssl.endpoint.identification.algorithm": "",
# "kafka.sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
}
The error is this: Timed out waiting for a node assignment.
