How much space does the metadata for a Delta table take up?

User16826992666
Databricks Employee
Databricks Employee

‎06-16-2021 07:41 PM

If you have a lot of transactions in a table it seems like the Delta log keeping track of all those transactions would get pretty large. Does the size of the metadata become a problem over time?

Labels:
0 Kudos

Ryan_Chynoweth
Databricks Employee
Databricks Employee

‎06-18-2021 02:07 PM

Yes, the size of the metadata can become a problem over time but not because of performance but because of storage costs. Delta performance will not degrade due to the size of the metadata, but your cloud storage bill can increase.

By default Delta history is only maintained for 30 days: https://docs.databricks.com/delta/delta-utility.html#retrieve-delta-table-history

This can be changed by setting the "delta.logRetentionDuration" to a longer period.

View solution in original post

0 Kudos