https://community.databricks.com/t5/data-engineering/streaming-kafka-data-without-duplication/m-p/81022#M36200 <P>Hello,</P><P>We are creating an application to read data from Kafka topic send by a source. After we get the data, we do some transformations and send to other Kafka topic. In this process source may send same data twice.</P><P>Our questions are</P><P>1. How can we control duplications and only send the updated data to target Kafka topic?</P><P>2. Where and what format should we store the data in Databricks to check for duplicates?</P><P>Thank You,</P><P>Dheeraj</P> Mon, 29 Jul 2024 22:06:00 GMT mddheeraj 2024-07-29T22:06:00Z https://community.databricks.com/t5/data-engineering/streaming-kafka-data-without-duplication/m-p/81022#M36200 <P>Hello,</P><P>We are creating an application to read data from Kafka topic send by a source. After we get the data, we do some transformations and send to other Kafka topic. In this process source may send same data twice.</P><P>Our questions are</P><P>1. How can we control duplications and only send the updated data to target Kafka topic?</P><P>2. Where and what format should we store the data in Databricks to check for duplicates?</P><P>Thank You,</P><P>Dheeraj</P> Mon, 29 Jul 2024 22:06:00 GMT https://community.databricks.com/t5/data-engineering/streaming-kafka-data-without-duplication/m-p/81022#M36200 mddheeraj 2024-07-29T22:06:00Z