cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Approach to monthly data snapshots

dpc
New Contributor III

Hello
I'm building a datawarehouse with all the usual facts and dimensions
It will flush (truncate) and rebuild on a monthly basis
Users have the need to not only view the data now but also view it historically i.e. what it was a point in time
My initial thought was to use time travel functionality but, am I right in saying that that's the wrong approach here, and we should really archive or apply a date filter to the data?

So, an alternate archiving strategy would be better?

 

Thanks

 

1 ACCEPTED SOLUTION

Accepted Solutions

Witold
Contributor III

This sounds like a SCD-2. Pick the approach which fits best for your use case.

FWIW, time traveling isn't supposed to be used for business queries, only for technical (e.g. select yesterday's changes). Due to the fact that historical entries are not persistent and will be removed ("vacuum") after a defined period.

View solution in original post

3 REPLIES 3

Witold
Contributor III

This sounds like a SCD-2. Pick the approach which fits best for your use case.

FWIW, time traveling isn't supposed to be used for business queries, only for technical (e.g. select yesterday's changes). Due to the fact that historical entries are not persistent and will be removed ("vacuum") after a defined period.

szymon_dybczak
Contributor III

Agree with Witold. Best practice is to use time travel for recoveries from issues between 7 and 30 days. (you can set much higher period, but it's not recommended).
Beyond that use snapshots for archival  history. 

dpc
New Contributor III

Great, thanks

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group