topic Kafka consumer and a web server simultaneously, thread blocking problem in microservice in Data Engineering https://community.databricks.com/t5/data-engineering/kafka-consumer-and-a-web-server-simultaneously-thread-blocking/m-p/16120#M10333 <P></P> <P><B>assumptions:</B> There are microservices behind an api-gateway, they communicate through HTTP synchronously. obviously, each one of those microservices is a web server. now I want my microservice to play as a Kafka producer and "consumer" too. more clearly, my microservice produces events and listens to some topics for another events.</P> <P><B>problem:</B> It seems traditionally a Kafka consumer use an infinite loop to poll the messages and stay alive (send heart-beats) my process thread is busy by serving host and cant be locked by an infinite loop. is there any way to somehow "listen" to the topic without block the thread just like pulsar listener or rabbit? I prefer not to separate my web server and consumer-processor due to the technical complexity I have to handle in development. may I use Kafka streams in this case? Is there something wrong with my assumptions?</P> <P></P> Mon, 23 Aug 2021 05:15:37 GMT reza-eghbali 2021-08-23T05:15:37Z Kafka consumer and a web server simultaneously, thread blocking problem in microservice https://community.databricks.com/t5/data-engineering/kafka-consumer-and-a-web-server-simultaneously-thread-blocking/m-p/16120#M10333 <P></P> <P><B>assumptions:</B> There are microservices behind an api-gateway, they communicate through HTTP synchronously. obviously, each one of those microservices is a web server. now I want my microservice to play as a Kafka producer and "consumer" too. more clearly, my microservice produces events and listens to some topics for another events.</P> <P><B>problem:</B> It seems traditionally a Kafka consumer use an infinite loop to poll the messages and stay alive (send heart-beats) my process thread is busy by serving host and cant be locked by an infinite loop. is there any way to somehow "listen" to the topic without block the thread just like pulsar listener or rabbit? I prefer not to separate my web server and consumer-processor due to the technical complexity I have to handle in development. may I use Kafka streams in this case? Is there something wrong with my assumptions?</P> <P></P> Mon, 23 Aug 2021 05:15:37 GMT https://community.databricks.com/t5/data-engineering/kafka-consumer-and-a-web-server-simultaneously-thread-blocking/m-p/16120#M10333 reza-eghbali 2021-08-23T05:15:37Z