https://community.databricks.com/t5/data-engineering/kafka-integration-with-databricks/m-p/19721#M13256 <P>This is code that I am using to read from Kafka</P><P></P><PRE><CODE>inputDF = (spark .readStream .format("kafka") .option("kafka.bootstrap.servers", host) .option("kafka.ssl.endpoint.identification.algorithm", "https") .option("kafka.sasl.mechanism", "PLAIN") .option("kafka.security.protocol", "SASL_SSL") .option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{}' password='{}';".format(userid, password)) .option("subscribe", topic) .option("kafka.client.id", "Databricks") .option("kafka.group.id", "new_group2") .option("spark.streaming.kafka.maxRatePerPartition", "5") .option("startingOffsets", "earliest") .option("kafka.session.timeout.ms", "10000") .option("minPartitions", sc.DefaultParallelism) .load() )</CODE></PRE><P></P> Wed, 30 Nov 2022 12:53:59 GMT Hubert-Dudek 2022-11-30T12:53:59Z https://community.databricks.com/t5/data-engineering/kafka-integration-with-databricks/m-p/19720#M13255 <P>Hi all</P><P>I want to integrate Kafka with databricks if anyone can share any doc or code it will help me a lot.</P><P>Thanks in advance</P> Wed, 30 Nov 2022 10:53:55 GMT https://community.databricks.com/t5/data-engineering/kafka-integration-with-databricks/m-p/19720#M13255 Ajay-Pandey 2022-11-30T10:53:55Z https://community.databricks.com/t5/data-engineering/kafka-integration-with-databricks/m-p/19721#M13256 <P>This is code that I am using to read from Kafka</P><P></P><PRE><CODE>inputDF = (spark .readStream .format("kafka") .option("kafka.bootstrap.servers", host) .option("kafka.ssl.endpoint.identification.algorithm", "https") .option("kafka.sasl.mechanism", "PLAIN") .option("kafka.security.protocol", "SASL_SSL") .option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{}' password='{}';".format(userid, password)) .option("subscribe", topic) .option("kafka.client.id", "Databricks") .option("kafka.group.id", "new_group2") .option("spark.streaming.kafka.maxRatePerPartition", "5") .option("startingOffsets", "earliest") .option("kafka.session.timeout.ms", "10000") .option("minPartitions", sc.DefaultParallelism) .load() )</CODE></PRE><P></P> Wed, 30 Nov 2022 12:53:59 GMT https://community.databricks.com/t5/data-engineering/kafka-integration-with-databricks/m-p/19721#M13256 Hubert-Dudek 2022-11-30T12:53:59Z https://community.databricks.com/t5/data-engineering/kafka-integration-with-databricks/m-p/19722#M13257 <P>Thanks @Hubert Dudek​&nbsp;</P> Thu, 01 Dec 2022 03:32:42 GMT https://community.databricks.com/t5/data-engineering/kafka-integration-with-databricks/m-p/19722#M13257 Kumarpandey 2022-12-01T03:32:42Z