topic Re: Can we implement Unity Catalog table lifecycle? in Data Engineering

Can we implement Unity Catalog table lifecycle?

Yuki — Fri, 25 Apr 2025 04:55:59 GMT

I want to delete tables that haven't been selected or otherwise accessed for several months.

I can see the Delta table history, but I can only catch the DDL or update/insert/delete and can't catch "select".
I realized that the Unity Catalog insight, https://docs.databricks.com/aws/en/discover/table-insights but I can't find the way to get data via SQL or API.

My idea is using Audit log, but it must consider security and get permissions from Admin team.
However, I'm thinking it is the best way to use audit log to achieve my purpose...

Is there any way to get or track history of all queries in Unity Catalog by each Schema or Tables?

Thank you.

Re: Can we implement Unity Catalog table lifecycle?

Renu_ — Fri, 25 Apr 2025 14:29:46 GMT

Hi @Yuki, I would say setting up audit logs is the best approach. You’ll need to work with your admin team to get this set up and make sure you have the right permissions, but it's the most reliable way to track table usage over time. Insights tab gives you a look at recent query activity, but it’s limited to the last 30 days and mostly focuses on saved queries in Databricks SQL.

Re: Can we implement Unity Catalog table lifecycle?

Yuki — Fri, 25 Apr 2025 15:21:28 GMT

Hi @Renu_ ,

I appreciate for your clear response. I now have a better understanding and will work with our admin team to develop a strategy.

Thank you.