topic Re: Why materialized views are created in __databricks_internal ? in Data Engineering

Why materialized views are created in __databricks_internal ?

AxelBrsn — Thu, 21 Mar 2024 09:40:32 GMT

Hello, I have a question about why materialized views are created in "__databricks_internal" catalog?

We specified catalog and schemas in the DLT Pipeline.

Re: Why materialized views are created in __databricks_internal ?

MathieuDB — Tue, 14 Jan 2025 15:09:53 GMT

Materialized views created by Delta Live Tables (DLT) pipelines are stored in the __databricks_internal catalog for several reasons:

Isolation: The __databricks_internal catalog is used to store system-generated tables, such as materializations, to prevent polluting the user-specified target catalog. Your specific catalog will only contains user-defined tables and views.
Ownership and Access Control: The materialization schema within the __databricks_internal catalog is owned by the system rather than the pipeline owner. This way it is easier to manage and the pipeline owner doesn't have to manage access controls to the underlying tables.
Governance: By storing materializations in a dedicated internal catalog, Databricks ensures that data governance and security policies are consistently applied.

Re: Why materialized views are created in __databricks_internal ?

Guntis — Thu, 19 Jun 2025 10:35:26 GMT

Hello @MathieuDB ,

I have follow-up question about __databricks_internal: what will happen if for AWS based workspace S3 bucket which is used by __databricks_internal catalog is deleted. Will it be recreated elsewhere by system user?

Kind regards,

Guntis.

Re: Why materialized views are created in __databricks_internal ?

MathieuDB — Thu, 19 Jun 2025 14:35:57 GMT

Hello Guntis,

This folder will not be recreated elsewhere if the bucket is deleted, as it is tied to the catalog. Could you please elaborate on the reason you'd like to delete this bucket in the first place?

Thanks

Re: Why materialized views are created in __databricks_internal ?

Yogesh_Verma_ — Thu, 19 Jun 2025 16:40:47 GMT

Hello,

Materialized views created by Delta Live Tables (DLT) are stored in the __databricks_internal catalog for a few key reasons:

Separation: This keeps system-generated tables (like materialized views) separate from your own tables and views, so your catalog stays clean and organized.
Access Control: These system tables are managed by Databricks, so you don’t need to worry about setting permissions or managing access.
Governance: Keeping materialized data in a dedicated internal catalog helps Databricks apply consistent security and data governance policies automatically.

Re: Why materialized views are created in __databricks_internal ?

Guntis — Wed, 25 Jun 2025 07:00:00 GMT

Hi, bucket could be deleted in case AWS account is decommissioned/deleted as a result of decommissioning Databricks workspace as there are other Databricks workspaces using the same Unity catalog metastore which will replace decommissioned workspace in different AWS accounts.