cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Anyone have any luck with Kafka? Any links, documents, videos that helped?

Chris_Shehu
Valued Contributor III

I'm trying to ingest streaming HL7 messages and planned to use Kafka with Smolder to achieve this. I'm having a little trouble understanding how Kafka can run on databricks. So far I've installed Kafka and started the cluster but the start command just spins forever. Am I crazy to think this should be possible? Any resources someone can share on how to accomplish this?

3 REPLIES 3

Debayan
Esteemed Contributor III

Thanks for reaching out to community.databricks.com.

Could you please refer this for the whole architecture: https://docs.databricks.com/structured-streaming/kafka.html

And also please refer this to know how to setup Apache Kafka on Databricks: https://kb.databricks.com/en_US/streaming/kafka-setup

jose_gonzalez
Moderator
Moderator

Hi @Christopher Shehuโ€‹,

Just a friendly follow-up. Did you have time to check Debayan's shared docs? do you still need help? please let us know

Chris_Shehu
Valued Contributor III

@Jose Gonzalezโ€‹  Sorry for the delay. I've actually already looked at the provided links. The first one is for AWS. I'm currently using Azure. The KAFKA setup is working..I think? I'm stuck at ok so I have a server listening now what? How do I send/retrieve messages? Do I need to configure some IP settings on the compute so it works properly in Azure? What do I provide to the interface server so it can send messages to the compute?

Thanks,

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group