https://community.databricks.com/t5/data-engineering/streaming-with-kafka-with-the-same-groupid/m-p/23350#M16099 <P>By default, each streaming query generates a unique group ID for reading data ( ensuring it's own &nbsp;its own consumer group&nbsp;) . In scenarios where you'd want to specify it (authz etc ) , it is not recommended to have two streaming applications specify the same groupid. Spark keeps track of Kafka offsets internally and doesn’t commit any offset.</P><P></P><P>In any case, for sources that doesn't support exactly once behaviour, with delta you could achieve idempotency via MERGE </P> Wed, 23 Jun 2021 05:26:23 GMT sajith_appukutt 2021-06-23T05:26:23Z https://community.databricks.com/t5/data-engineering/streaming-with-kafka-with-the-same-groupid/m-p/23349#M16098 <P>A kafka topic is having 300 partitions and I see two clusters are running and have the same group id, </P><P></P><P> will the data be duplicate in my delta bonze layer</P> Thu, 17 Jun 2021 08:34:31 GMT https://community.databricks.com/t5/data-engineering/streaming-with-kafka-with-the-same-groupid/m-p/23349#M16098 User16826994223 2021-06-17T08:34:31Z https://community.databricks.com/t5/data-engineering/streaming-with-kafka-with-the-same-groupid/m-p/23350#M16099 <P>By default, each streaming query generates a unique group ID for reading data ( ensuring it's own &nbsp;its own consumer group&nbsp;) . In scenarios where you'd want to specify it (authz etc ) , it is not recommended to have two streaming applications specify the same groupid. Spark keeps track of Kafka offsets internally and doesn’t commit any offset.</P><P></P><P>In any case, for sources that doesn't support exactly once behaviour, with delta you could achieve idempotency via MERGE </P> Wed, 23 Jun 2021 05:26:23 GMT https://community.databricks.com/t5/data-engineering/streaming-with-kafka-with-the-same-groupid/m-p/23350#M16099 sajith_appukutt 2021-06-23T05:26:23Z