duplicate data published in kafka offset

dipali_globant — Tue, 29 Oct 2024 14:24:37 GMT

we have 25k data which are publishing by batch of 5k.

we are numbering the records based on row_number window function and creating batch using this.

we have observed that some records like 10-20 records are getting published duplicated in 2 offset.

can someone help with the probable cause for this issue.

Re: duplicate data published in kafka offset

agallard — Tue, 29 Oct 2024 15:54:34 GMT

Hi @dipali_globant,

duplicate data in Kafka can arise in a batch processing scenario for a few reasons 🤔

here’s an example of ensuring unique and consistent row numbering:

from pyspark.sql import Window from pyspark.sql.functions import row_number window_spec = Window.orderBy("unique_column") # Replace "unique_column" with a reliable ordering column df = df.withColumn("row_number", row_number().over(window_spec))

The ROW_NUMBER window function may assign different row numbers across batch executions if there’s no consistent order. This can result in the same records being included in multiple batches, especially if partitions or order change between runs, so you can use a specific column or combination of columns for ordering that is unique and stable (such as timestamps or primary keys).

If you are processing and publishing multiple batches simultaneously, concurrency issues could arise, especially if some data falls between two consecutive batch windows.

check and comments!

Regards

topic duplicate data published in kafka offset in Get Started Discussions

duplicate data published in kafka offset

Re: duplicate data published in kafka offset