topic Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks in Data Engineering

Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks

selvakumar092 — Thu, 15 Jun 2023 08:48:18 GMT

I am trying to do incremental load in azure data factory. Most of the tables in the Oracle database doesn't have last modified date and Primary key column. Is there any way to do incremental loading without last modified date and primary key column?

Re: Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks

-werners- — Thu, 15 Jun 2023 09:19:36 GMT

ouch, that is a tough cookie.

If you have a cdc-system in place, you could extract from that (although you first have to be able to connect it to ADF too). Otherwise this will become very hard.

If the amount of records in the concerning tables is not that big, I would not bother going for incremental.

Re: Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks

selvakumar092 — Thu, 15 Jun 2023 09:35:28 GMT

Hi @Werner Stinckens ,

Thanks for your response. The data volume of the table is huge, so can't do always full load. The Oracle data source which I mentioned was all the tables are views. We don't not have the CDC in place already. Is there any alternative tools (like Fivetran) which help us to solve this?

Re: Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks

-werners- — Thu, 15 Jun 2023 09:54:43 GMT

To work incrementally you either need some kind of increasing key or some system that registers CRUD operations on the database, be it Fivetran or something else.

The increasing key only tackles the inserts, and you mention you do not have such a thing.

So I'd either add a lastmod column or install some software that registers changes on Oracle (f.e. a log-based cdc system).

Re: Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks

selvakumar092 — Thu, 15 Jun 2023 10:20:28 GMT

Thank you so much, will look in to this. 👍

Re: Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks

Anonymous — Fri, 16 Jun 2023 03:16:42 GMT

Hi @Selva Kumar Ponnusamy

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!