topic Re: Glue Catalog Metadata Management with Enforced Tagging in Databricks in Data Engineering

Glue Catalog Metadata Management with Enforced Tagging in Databricks

KartRasi_10779 — Wed, 16 Oct 2024 17:01:56 GMT

As part of the data governance team, we're trying to enforce table-level tagging when users create tables in a Databricks environment where metadata is managed by AWS Glue Catalog (non-Unity Catalog). Is there a way to require tagging at table creation, or to set up a process that enforces tagging in such an environment? Any guidance on best practices or tools to implement this would be appreciated

Re: Glue Catalog Metadata Management with Enforced Tagging in Databricks

Brahmareddy — Wed, 23 Oct 2024 16:02:04 GMT

Hi @KartRasi_10779,

How are you doing today?

As per my understanding, Consider creating a tagging enforcement process by implementing a custom script or trigger that checks for table tags post-creation, as AWS Glue Catalog doesn't natively support mandatory tagging at table creation. You could use AWS Lambda functions combined with Glue triggers to monitor for new table creation and automatically apply or validate tags. Another approach is to integrate Databricks jobs that run periodically to ensure all tables have the required tags and alert when tags are missing. Best practices would include setting up clear tagging policies and using automation tools like Databricks REST API or Glue Crawlers to ensure tags are consistently applied across all tables.

Give a try and let me know the outcome.

Hoping you have a good day.

Regards,

Brahma

Re: Glue Catalog Metadata Management with Enforced Tagging in Databricks

145676 — Mon, 28 Oct 2024 20:48:51 GMT

You can use lakeFS pre-merge hooks to force this. Works great with this stack -> https://lakefs.io/blog/lakefs-hooks/