Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Materialized views created by Delta Live Tables (DLT) pipelines are stored in the __databricks_internal catalog for several reasons:
Isolation: The __databricks_internal catalog is used to store system-generated tables, such as materializations, to prevent polluting the user-specified target catalog. Your specific catalog will only contains user-defined tables and views.
Ownership and Access Control: The materialization schema within the __databricks_internal catalog is owned by the system rather than the pipeline owner. This way it is easier to manage and the pipeline owner doesn't have to manage access controls to the underlying tables.
Governance: By storing materializations in a dedicated internal catalog, Databricks ensures that data governance and security policies are consistently applied.
Join Us as a Local Community Builder!
Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!
