topic Re: Anyone have any luck with Kafka? Any links, documents, videos that helped? in Data Engineering

Anyone have any luck with Kafka? Any links, documents, videos that helped?

Chris_Shehu — Fri, 02 Sep 2022 17:34:43 GMT

I'm trying to ingest streaming HL7 messages and planned to use Kafka with Smolder to achieve this. I'm having a little trouble understanding how Kafka can run on databricks. So far I've installed Kafka and started the cluster but the start command just spins forever. Am I crazy to think this should be possible? Any resources someone can share on how to accomplish this?

Re: Anyone have any luck with Kafka? Any links, documents, videos that helped?

jose_gonzalez — Fri, 09 Sep 2022 23:19:34 GMT

Hi @Christopher Shehu,

Just a friendly follow-up. Did you have time to check Debayan's shared docs? do you still need help? please let us know

Re: Anyone have any luck with Kafka? Any links, documents, videos that helped?

Chris_Shehu — Thu, 22 Sep 2022 16:12:27 GMT

@Jose Gonzalez Sorry for the delay. I've actually already looked at the provided links. The first one is for AWS. I'm currently using Azure. The KAFKA setup is working..I think? I'm stuck at ok so I have a server listening now what? How do I send/retrieve messages? Do I need to configure some IP settings on the compute so it works properly in Azure? What do I provide to the interface server so it can send messages to the compute?

Thanks,

Re: Anyone have any luck with Kafka? Any links, documents, videos that helped?

Debayan — Fri, 02 Sep 2022 20:21:29 GMT

Thanks for reaching out to community.databricks.com.

Could you please refer this for the whole architecture: https://docs.databricks.com/structured-streaming/kafka.html

And also please refer this to know how to setup Apache Kafka on Databricks: https://kb.databricks.com/en_US/streaming/kafka-setup