cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

elgeo
by Valued Contributor II
  • 3020 Views
  • 3 replies
  • 5 kudos

Resolved! Delta Table - Reduce time travel storage size

Hello! I am trying to understand time travel feature. I see with "DESCRIBE HISTORY" command that all the transaction history on a specific table is recorded by version and timestamp. However, I understand that this occupies a lot of storage especiall...

  • 3020 Views
  • 3 replies
  • 5 kudos
Latest Reply
elgeo
Valued Contributor II
  • 5 kudos

Thank you @Werner Stinckens​ for your reply. However I still haven't managed to delete history even after setting the below. The number of history rows remains the same when running "DESCRIBE HISTORY".SET spark.databricks.delta.retentionDurationCheck...

  • 5 kudos
2 More Replies
Gim
by Contributor
  • 6011 Views
  • 7 replies
  • 10 kudos

Resolved! Delta Table storage best practices

Hi!We have a project where we do some Data Engineering for a client. I implemented a scheduled batch processing of Databricks' autoloader (stream w/ availableNow) since they primarily have numerous file exports from several sources. We wanted to foll...

  • 6011 Views
  • 7 replies
  • 10 kudos
Latest Reply
jose_gonzalez
Moderator
  • 10 kudos

Hi @Gimwell Young​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 10 kudos
6 More Replies
Labels