cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day.

vinaykumar
New Contributor III

Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day.

 For example , let’s assume we are doing multiple operation on table in a day every minute and want to keep time travel history for each day of latest version of a day for one week .

1 ACCEPTED SOLUTION

Accepted Solutions

Lakshay
Databricks Employee
Databricks Employee

If I understood correctly, you want to keep only one version per day regardless of the number of transactions done on that day. The problem in achieving this is that if you delete any of the versions in a delta table, you will lose the ability to time travel to any version before that.

The only way I think this is achievable is by keeping some sort of record(in a separate table perhaps) of the latest version for each day and use information from there to do a time travel

View solution in original post

3 REPLIES 3

Lakshay
Databricks Employee
Databricks Employee

If I understood correctly, you want to keep only one version per day regardless of the number of transactions done on that day. The problem in achieving this is that if you delete any of the versions in a delta table, you will lose the ability to time travel to any version before that.

The only way I think this is achievable is by keeping some sort of record(in a separate table perhaps) of the latest version for each day and use information from there to do a time travel

Anonymous
Not applicable

Hi @vinay kumar​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

vinaykumar
New Contributor III

Yes , Issue got resolved . Thanks

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group