Anyone have any luck with Kafka? Any links, documents, videos that helped?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-02-2022 10:34 AM
I'm trying to ingest streaming HL7 messages and planned to use Kafka with Smolder to achieve this. I'm having a little trouble understanding how Kafka can run on databricks. So far I've installed Kafka and started the cluster but the start command just spins forever. Am I crazy to think this should be possible? Any resources someone can share on how to accomplish this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-02-2022 01:21 PM
Thanks for reaching out to community.databricks.com.
Could you please refer this for the whole architecture: https://docs.databricks.com/structured-streaming/kafka.html
And also please refer this to know how to setup Apache Kafka on Databricks: https://kb.databricks.com/en_US/streaming/kafka-setup
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-09-2022 04:19 PM
Hi @Christopher Shehu,
Just a friendly follow-up. Did you have time to check Debayan's shared docs? do you still need help? please let us know
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-22-2022 09:12 AM
@Jose Gonzalez Sorry for the delay. I've actually already looked at the provided links. The first one is for AWS. I'm currently using Azure. The KAFKA setup is working..I think? I'm stuck at ok so I have a server listening now what? How do I send/retrieve messages? Do I need to configure some IP settings on the compute so it works properly in Azure? What do I provide to the interface server so it can send messages to the compute?
Thanks,