Why materialized views are created in __databricks_internal ?

[AxelBrsn]

Hello, I have a question about why materialized views are created in "__databricks_internal" catalog?

We specified catalog and schemas in the DLT Pipeline.

[MathieuDB]

Hello @AxelBrsn 

Materialized views created by Delta Live Tables (DLT) pipelines are stored in the __databricks_internal catalog for several reasons:

1. Isolation: The __databricks_internal catalog is used to store system-generated tables, such as materializations, to prevent polluting the user-specified target catalog. Your specific catalog will only contains user-defined tables and views.
   
   
2. Ownership and Access Control: The materialization schema within the __databricks_internal catalog is owned by the system rather than the pipeline owner. This way it is easier to manage and the pipeline owner doesn't have to manage access controls to the underlying tables.
   
   
3. Governance: By storing materializations in a dedicated internal catalog, Databricks ensures that data governance and security policies are consistently applied.

[Guntis]

Hello @MathieuDB ,

I have follow-up question about __databricks_internal: what will happen if for AWS based workspace S3 bucket which is used by __databricks_internal catalog is deleted. Will it be recreated elsewhere by system user?

Kind regards,

Guntis.

[MathieuDB]

Hello Guntis,

This folder will not be recreated elsewhere if the bucket is deleted, as it is tied to the catalog. Could you please elaborate on the reason you'd like to delete this bucket in the first place?

Thanks 

[Guntis]

Hi, bucket could be deleted in case AWS account is decommissioned/deleted as a result of decommissioning Databricks workspace as there are other Databricks workspaces using the same Unity catalog metastore which will replace decommissioned workspace in different AWS accounts.

[Yogesh_378691]

Hello,

Materialized views created by Delta Live Tables (DLT) are stored in the __databricks_internal catalog for a few key reasons:

• Separation: This keeps system-generated tables (like materialized views) separate from your own tables and views, so your catalog stays clean and organized.

• Access Control: These system tables are managed by Databricks, so you don’t need to worry about setting permissions or managing access.

• Governance: Keeping materialized data in a dedicated internal catalog helps Databricks apply consistent security and data governance policies automatically.