Hi All, I'm looking to implement an automated, scalable, and auditable purge mechanism on Azure Databricks to manage data retention, deletion and archival policies across our Unity Catalog-governed Delta tables.

I've come across various approaches, such as using VACUUM, REORG, and custom scripts, but I'd like to know if there are any best practices or recommended solutions for implementing such a mechanism.

Specifically, I'm interested in learning about: across all layers (Bronze, Silver, and Gold) for retention ,deletion and archival policies

I'd appreciate any guidance, advice, or best practices you can share on implementing an automated, scalable, and auditable purge mechanism on Azure Databricks.

Regards,

Phani