Unity Catalog

Bhuvnesh — Mon, 28 Oct 2024 09:33:54 GMT

Hi,

I have requirement to setup the athena tables. We have a unity catalog setup in databricks workspace and I would like know is there any possibility that Athen can be point to unity catalog so that all the tables are available in athena.

whenever we update or create any new table in unity catalog it gets updated in aws athena as well.

i'm interested in available options

Thanks

Re: Unity Catalog

ArunKhandelwal — Mon, 28 Oct 2024 09:58:35 GMT

Unfortunately, as of now, there isn't a direct, seamless integration between Unity Catalog and Athena to automatically synchronize table updates.

However, here are a few potential approaches to achieve your desired outcome:

1. AWS Glue Data Catalog:

Manual Synchronization:
- Create a Glue Crawler to scan the S3 location where your Delta Lake tables are stored.
- Configure the crawler to update the Glue Data Catalog periodically.
- Athena can then query the Glue Data Catalog to access the latest table definitions.
Semi-Automated Synchronization:
- Use a scripting approach (Python, Scala) to trigger the Glue Crawler whenever changes are made to Unity Catalog.

2. Databricks Delta Sharing:

Share Delta Tables:
- Share your Delta tables from Databricks with external users or applications.
- Configure Athena to access these shared Delta tables directly.
- This approach provides a more seamless integration but requires careful management of access controls and data security.

3. Custom Connectors or APIs:

Develop a Custom Connector:
- Build a custom connector to integrate Athena with Databricks Unity Catalog.
- This approach requires significant development effort and may not be feasible for all use cases.
Use APIs:
- Leverage the Databricks REST API to retrieve metadata about tables and schemas from Unity Catalog.
- This information can then be used to update the Glue Data Catalog or create custom Athena views

topic Re: Unity Catalog in Data Engineering

Unity Catalog

Re: Unity Catalog