In most Lakehouse implementations, teams (mine included) have traditionally used Schemas to segregate tables for the Medallion ArchitectureโBronze, Silver, and Gold. However, a major challenge was that when declaring a catalog and schema in Delta Live Tables (DLT), all tables and views had to be registered in the same schema.
This meant either:
- Keeping all tables from Bronze to Gold in one schemaโresulting in messy, unorganized data.
- Creating multiple separate pipelines just to maintain proper segregation
- Inability to tack pipeline progress and dependencies via the graph
But now, with this new Databricks release, users can register tables to different catalogs and schemas in Unity Catalog!
Key Highlights:
Automatic Enablement for New Pipelines: Any new pipelines created through the UI will have this functionality enabled by default (if the preview is enabled). However, existing pipelines cannot be converted to utilize this feature. To migrate, you must create a new pipeline and re-ingest data from the source(s).
- Fully Qualified Table Names for Streaming Tables: When creating streaming tables or views, you must register them using the fully qualified path (catalog.schema.table). This ensures that subsequent processes can accurately read the data.
- Schema Configuration: Because you have to use the full qualified path, the schema where the pipeline is defined cannot be the same as the schema where the tables are registered. Attempting to register tables in the same schema will result in an error.
- Pipeline Creation via API: If you're creating the pipeline using the API, to register it as a Unity Catalog Pipeline, you need to use the schema object in your JSON configuration instead of the previous target parameter. Using target will configure the pipeline in the old mode.
Why This Matters:
- Maintain a clean Medallion Architecture without the need for multiple pipelines.
- Organize tables into separate schemas for enhanced governance.
- Simplify access control and multi-environment management.
How do you plan to leverage this update? Let's discuss how this can enhance data governance and streamline your pipeline designs!
