Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Is there a way to make audit on all tables in hive_metastore (no UC), all are external, to check when each has been used for the last time (queried / updated / etc). ?
Apache Ranger or Apache Sentry can be used for auditing Hive activities. If you have set up auditing in one of these tools, you can review the audit logs to see when tables were accessed. Audit logs are typically stored in a separate location, and you'll need to refer to the documentation of the specific tool you are using for more details. You can modify your Hive queries or scripts to log information about table access to a custom log file. This would involve adding logging statements in your Hive scripts or applications.
Thank you for the suggestions, will check both. However for the hive scripts, it's near impossible, as tables are queried ad-hoc from notebooks/ created ad-hoc. but noone is doing any cleanup and I feel audit is very needed ๐
Join 100K+ Data Experts: Register Now & Grow with Us!
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!
