cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day.

vinaykumar
New Contributor III

Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day.

 For example , letโ€™s assume we are doing multiple operation on table in a day every minute and want to keep time travel history for each day of latest version of a day for one week .

1 ACCEPTED SOLUTION

Accepted Solutions

Lakshay
Esteemed Contributor
Esteemed Contributor

If I understood correctly, you want to keep only one version per day regardless of the number of transactions done on that day. The problem in achieving this is that if you delete any of the versions in a delta table, you will lose the ability to time travel to any version before that.

The only way I think this is achievable is by keeping some sort of record(in a separate table perhaps) of the latest version for each day and use information from there to do a time travel

View solution in original post

3 REPLIES 3

Lakshay
Esteemed Contributor
Esteemed Contributor

If I understood correctly, you want to keep only one version per day regardless of the number of transactions done on that day. The problem in achieving this is that if you delete any of the versions in a delta table, you will lose the ability to time travel to any version before that.

The only way I think this is achievable is by keeping some sort of record(in a separate table perhaps) of the latest version for each day and use information from there to do a time travel

Anonymous
Not applicable

Hi @vinay kumarโ€‹ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

vinaykumar
New Contributor III

Yes , Issue got resolved . Thanks

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.