Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks

[selvakumar092]

 I am trying to do incremental load in azure data factory. Most of the tables in the Oracle database doesn't have last modified date and Primary key column. Is there any way to do incremental loading without last modified date and primary key column?

[-werners-]

ouch, that is a tough cookie.

If you have a cdc-system in place, you could extract from that (although you first have to be able to connect it to ADF too). Otherwise this will become very hard.

If the amount of records in the concerning tables is not that big, I would not bother going for incremental.

[selvakumar092]

Hi @Werner Stinckens​ ,

Thanks for your response. The data volume of the table is huge, so can't do always full load. The Oracle data source which I mentioned was all the tables are views. We don't not have the CDC in place already. Is there any alternative tools (like Fivetran) which help us to solve this?

[-werners- · Accepted solution]

To work incrementally you either need some kind of increasing key or some system that registers CRUD operations on the database, be it Fivetran or something else.

The increasing key only tackles the inserts, and you mention you do not have such a thing.

So I'd either add a lastmod column or install some software that registers changes on Oracle (f.e. a log-based cdc system).

[selvakumar092]

Thank you so much, will look in to this. 👍

[Anonymous]

Hi @Selva Kumar Ponnusamy​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!