https://community.databricks.com/t5/data-engineering/how-much-space-does-the-metadata-for-a-delta-table-take-up/m-p/23576#M16298 <P>Yes, the size of the metadata can become a problem over time but not because of performance but because of storage costs. Delta performance will not degrade due to the size of the metadata, but your cloud storage bill can increase. </P><P></P><P>By default Delta history is only maintained for 30 days: <A href="https://docs.databricks.com/delta/delta-utility.html#retrieve-delta-table-history" target="test_blank">https://docs.databricks.com/delta/delta-utility.html#retrieve-delta-table-history</A></P><P></P><P>This can be changed by setting the "delta.logRetentionDuration" to a longer period. </P> Fri, 18 Jun 2021 21:07:09 GMT Ryan_Chynoweth 2021-06-18T21:07:09Z https://community.databricks.com/t5/data-engineering/how-much-space-does-the-metadata-for-a-delta-table-take-up/m-p/23575#M16297 <P>If you have a lot of transactions in a table it seems like the Delta log keeping track of all those transactions would get pretty large. Does the size of the metadata become a problem over time?</P> Thu, 17 Jun 2021 02:41:25 GMT https://community.databricks.com/t5/data-engineering/how-much-space-does-the-metadata-for-a-delta-table-take-up/m-p/23575#M16297 User16826992666 2021-06-17T02:41:25Z https://community.databricks.com/t5/data-engineering/how-much-space-does-the-metadata-for-a-delta-table-take-up/m-p/23576#M16298 <P>Yes, the size of the metadata can become a problem over time but not because of performance but because of storage costs. Delta performance will not degrade due to the size of the metadata, but your cloud storage bill can increase. </P><P></P><P>By default Delta history is only maintained for 30 days: <A href="https://docs.databricks.com/delta/delta-utility.html#retrieve-delta-table-history" target="test_blank">https://docs.databricks.com/delta/delta-utility.html#retrieve-delta-table-history</A></P><P></P><P>This can be changed by setting the "delta.logRetentionDuration" to a longer period. </P> Fri, 18 Jun 2021 21:07:09 GMT https://community.databricks.com/t5/data-engineering/how-much-space-does-the-metadata-for-a-delta-table-take-up/m-p/23576#M16298 Ryan_Chynoweth 2021-06-18T21:07:09Z