https://community.databricks.com/t5/get-started-discussions/parallel-kafka-consumer-in-spark-structured-streaming/m-p/70276#M7277 <P>Hi,</P><P>I have a spark streaming job which reads from kafka and process data and write to delta lake.</P><P>Number of kafka partition: 100</P><P>number of executor: 2 (4 core each)</P><P>So we have 8 cores total which are reading from 100 partitions of a topic. I wanted to understand if spark internally spin up muliple threads to reads from multiple partitions in parallel? if not is there any way to spin up multiple threads for kafka consumer.</P> Wed, 22 May 2024 14:29:11 GMT subham0611 2024-05-22T14:29:11Z https://community.databricks.com/t5/get-started-discussions/parallel-kafka-consumer-in-spark-structured-streaming/m-p/70276#M7277 <P>Hi,</P><P>I have a spark streaming job which reads from kafka and process data and write to delta lake.</P><P>Number of kafka partition: 100</P><P>number of executor: 2 (4 core each)</P><P>So we have 8 cores total which are reading from 100 partitions of a topic. I wanted to understand if spark internally spin up muliple threads to reads from multiple partitions in parallel? if not is there any way to spin up multiple threads for kafka consumer.</P> Wed, 22 May 2024 14:29:11 GMT https://community.databricks.com/t5/get-started-discussions/parallel-kafka-consumer-in-spark-structured-streaming/m-p/70276#M7277 subham0611 2024-05-22T14:29:11Z