brickster_2018
Databricks Employee
Databricks Employee

To time travel to a previous version, you need data and metadata. Metadata(json files in the delta log directory) by default comes with a retention of 30 days. You will need to increase the retention to be able to time travel to older versions. (delta.logRetentionDuration)

Similarly, you need to increase the time your old data files should be stored(delta.deletedFileRetentionDuration)

Configuration details : https://docs.databricks.com/delta/delta-batch.html#data-retention

Read more about Delta log structure here - https://github.com/delta-io/delta/blob/master/PROTOCOL.md

View solution in original post