https://community.databricks.com/t5/data-engineering/duplicates-detected-in-transformed-data-help-with/m-p/125267#M47395 <P>Hello</P><P>Can anyone help with an error I am getting when running ADF. An ingestion pipeline fails and when I click on the link I am taken to a Databricks error message "7 duplicates detected in transformed data". However, when I run the transformation cell of the notebook in question I get no issues with the data produced and there are zero duplicate rows. Another notebook referencing this notebook (which is also run as part of the ADF pipeline) has a check for duplicates and that is what is causing the ADF ingestion pipeline to fail. Since I have been unable to replicate the error and identify any duplicate rows based on the SQL which is being run in the Databricks notebook, is anyone able to advise me on anything I can do within Databricks to get it to tell me what the 7 rows of data in question are? Sorry if this request is a bit muddled, I am new to Databricks.</P><P>Thank you</P> Tue, 15 Jul 2025 09:45:19 GMT Firehose74 2025-07-15T09:45:19Z https://community.databricks.com/t5/data-engineering/duplicates-detected-in-transformed-data-help-with/m-p/125267#M47395 <P>Hello</P><P>Can anyone help with an error I am getting when running ADF. An ingestion pipeline fails and when I click on the link I am taken to a Databricks error message "7 duplicates detected in transformed data". However, when I run the transformation cell of the notebook in question I get no issues with the data produced and there are zero duplicate rows. Another notebook referencing this notebook (which is also run as part of the ADF pipeline) has a check for duplicates and that is what is causing the ADF ingestion pipeline to fail. Since I have been unable to replicate the error and identify any duplicate rows based on the SQL which is being run in the Databricks notebook, is anyone able to advise me on anything I can do within Databricks to get it to tell me what the 7 rows of data in question are? Sorry if this request is a bit muddled, I am new to Databricks.</P><P>Thank you</P> Tue, 15 Jul 2025 09:45:19 GMT https://community.databricks.com/t5/data-engineering/duplicates-detected-in-transformed-data-help-with/m-p/125267#M47395 Firehose74 2025-07-15T09:45:19Z https://community.databricks.com/t5/data-engineering/duplicates-detected-in-transformed-data-help-with/m-p/129796#M48606 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/139430">@Firehose74</a>&nbsp;,</P> <P>This may need a deeper investigation and require workspace access to troubleshoot/review logs. Can you please raise a ticket with us?</P> Tue, 26 Aug 2025 10:29:28 GMT https://community.databricks.com/t5/data-engineering/duplicates-detected-in-transformed-data-help-with/m-p/129796#M48606 Sidhant07 2025-08-26T10:29:28Z