Unity catalog accounts

[Phani1]

Hi Team,

We have the requirement to have metadata(Unity catalog) in one AWS account and data storage(Delta tables under data) in another account, is it possible to do that , Do we face any technical/Security issue??

[Kaniz]

Hi @Phani1, Let’s address your requirement regarding Unity Catalog metadata and Delta tables storage in separate AWS accounts.

Unity Catalog Accounts:

• Unity Catalog (UC) is a fine-grained governance solution for data and AI on the Databricks Lakehouse.
• UC manages metadata, lineage information, and other governance aspects.
• Each UC metastore requires a Storage Account container path where UC-managed table data is stored.
• UC delta tables’ table metadata reside inside this storage account, specifically in a directory named __unitystorage.
• Other metadata, such as UC lineage info, is stored in the regional Control Plane1.

Separate AWS Accounts:

• Technically, it is possible to have metadata (Unity catalog) in one AWS account and data storage (Delta tables) in another.
• However, there are considerations related to security, networking, and access control:
  • Security: Ensure proper IAM (Identity and Access Management) policies and permissions between the two accounts.
  • Networking: Set up appropriate VPC (Virtual Private Cloud) peering or transit gateway connections.
  • Access Control: Define fine-grained access controls to restrict cross-account access.
• Best Practice: Follow Unity Catalog best practices to simplify security and governance of your data2.

Technical/Security Issues:

• While technically feasible, it’s essential to evaluate the following:
  • Data Consistency: Ensure consistency between metadata and data storage.
  • Latency: Cross-account communication may introduce latency.
  • Auditability: Maintain audit logs for cross-account interactions.
  • Compliance: Consider any regulatory or compliance requirements.
• Collaborate with your AWS and Databricks teams to address these concerns.