cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks

selvakumar092
New Contributor II

 I am trying to do incremental load in azure data factory. Most of the tables in the Oracle database doesn't have last modified date and Primary key column. Is there any way to do incremental loading without last modified date and primary key column?

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

To work incrementally you either need some kind of increasing key or some system that registers CRUD operations on the database, be it Fivetran or something else.

The increasing key only tackles the inserts, and you mention you do not have such a thing.

So I'd either add a lastmod column or install some software that registers changes on Oracle (f.e. a log-based cdc system).

View solution in original post

5 REPLIES 5

-werners-
Esteemed Contributor III

ouch, that is a tough cookie.

If you have a cdc-system in place, you could extract from that (although you first have to be able to connect it to ADF too). Otherwise this will become very hard.

If the amount of records in the concerning tables is not that big, I would not bother going for incremental.

Hi @Werner Stinckens​ ,

Thanks for your response. The data volume of the table is huge, so can't do always full load. The Oracle data source which I mentioned was all the tables are views. We don't not have the CDC in place already. Is there any alternative tools (like Fivetran) which help us to solve this?

-werners-
Esteemed Contributor III

To work incrementally you either need some kind of increasing key or some system that registers CRUD operations on the database, be it Fivetran or something else.

The increasing key only tackles the inserts, and you mention you do not have such a thing.

So I'd either add a lastmod column or install some software that registers changes on Oracle (f.e. a log-based cdc system).

Thank you so much, will look in to this. 👍

Anonymous
Not applicable

Hi @Selva Kumar Ponnusamy​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!