Materialized View backing pipeline retains old Unity Catalog after catalog rename

CURIOUS_DE
Valued Contributor

yesterday


Materialized View backing pipeline retains old Unity Catalog after catalog rename (dev_cat → dap_catalog_dev) resulting in NO_SUCH_CATALOG_EXCEPTION

This issue blocks the creation and refresh of Materialized Views after a Unity Catalog rename.

Workspace Information

Pipeline Run ID

 
5337905b-2d28-45f0-8f40-b9279943436a

Cluster used during validation

 
0722-134910-ls3f4px7-v2n

Problem Summary

We renamed our Unity Catalog from:

 
dev_cat

to

 

dap_catalog_dev

using the Databricks UI.

Following the rename, existing Materialized Views became invalid.

Even after dropping the Materialized View and recreating it in the renamed catalog, Databricks continues referencing the old catalog (dev_cat) while validating the backing managed pipeline.

As a result, creation/validation of the Materialized View fails.

Error

Encountered an error with Unity Catalog while setting up the pipeline on cluster 0722-134910-ls3f4px7-v2n.

Ensure that your Unity Catalog configuration is correct, and that required resources (e.g., catalog, schema) exist and are accessible.

Also verify that the cluster has appropriate permissions to access Unity Catalog.

Details:

[NO_SUCH_CATALOG_EXCEPTION]
Catalog 'dev_cat' was not found.

SQLSTATE: 42704

Environment

  • Unity Catalog enabled workspace
  • Materialized Views
  • Serverless Pipeline
  • Managed Pipeline generated by Materialized View
  • Pipeline uses Photon
  • User has Workspace Admin privileges

Steps Performed

Step 1

Originally created Materialized View under

 

 
dev_cat

Everything functioned correctly.

Step 2

Renamed catalog

 
dev_cat

to

 
dap_catalog_dev

using the Databricks UI.

Step 3

Dropped the original Materialized View.

The SQL object was successfully removed.

However, the backing managed pipeline still exists.

Step 4

Attempted to recreate the Materialized View under

 

 
dap_catalog_dev

using

 

 
CREATE OR REPLACE MATERIALIZED VIEWdap_catalog_dev.caltrack.vw_item_17_electric_hourly_materializedAS...

Pipeline validation immediately fails.

Investigation Performed

We inspected the generated pipeline YAML.

The managed pipeline still contains references to the old catalog.

Pipeline Name

 

 
MV-dev_cat.caltrack.vw_item_17_electric_hourly_materialized

Pipeline configuration contains

 
catalog: dev_cat

Managed SQL definition

 
CREATE MATERIALIZED VIEWdev_cat.caltrack.vw_item_17_electric_hourly_materialized

However, the source dataset references the renamed catalog

 

 
FROMdap_catalog_dev.caltrack.vw_item_17_electric_hourly

Additionally,

 
dataset_name:
dap_catalog_dev.caltrack.vw_item_17_electric_hourly_materialized

This results in inconsistent metadata.

Summary of pipeline metadata:

Property Value
Pipeline NameMV-dev_cat.caltrack.vw_item_17_electric_hourly_materialized
catalogdev_cat
CREATE MATERIALIZED VIEW targetdev_cat
Source Viewdap_catalog_dev
dataset_namedap_catalog_dev

Additional Observation

Running

 
DESCRIBE EXTENDED
dap_catalog_dev.caltrack.vw_item_17_electric_yearly_agg_materialized;

returns

TABLE_DOES_NOT_EXIST

Table

dev_cat.caltrack.__materialization_mat_...

does not exist

This indicates the hidden internal materialization table still references the old catalog.

Expected Behaviour

After renaming a Unity Catalog,

or after dropping and recreating the Materialized View,

the managed pipeline should be regenerated with

 

 
catalog = dap_catalog_dev

and all internal metadata should reference the renamed catalog.

Actual Behaviour

The managed pipeline continues using

 
catalog = dev_cat

which no longer exists.

Validation therefore fails with

 

 
NO_SUCH_CATALOG_EXCEPTION

Troubleshooting Already Performed

  • Confirmed catalog rename completed successfully.
  • Confirmed new catalog exists.
  • Confirmed schemas exist.
  • Confirmed user has administrative permissions.
  • Dropped the Materialized View.
  • Waited for cleanup.
  • Recreated the Materialized View.
  • Verified pipeline still exists.
  • Inspected generated pipeline YAML.
  • Verified stale catalog references.
  • Verified hidden materialization table still references the old catalog.
Databricks Solution Architect
0 Kudos

iyashk-DB
Databricks Employee
Databricks Employee

yesterday

What you're hitting is expected behavior, not something you did wrong. Every materialized view is backed by a managed serverless pipeline, and that pipeline pins its catalog, target schema, and internal storage location at the moment the MV is created. A catalog rename updates the catalog entry in Unity Catalog, but it does not rewrite the config baked into an already-existing backing pipeline or its hidden materialization tables. So the pipeline keeps looking for dev_cat, which no longer exists, and you get NO_SUCH_CATALOG_EXCEPTION.

The reason your drop-and-recreate didn't fully clear it: dropping the MV is supposed to tear down the managed pipeline behind it, but when the pipeline is already in a broken state pointing at a missing catalog, the old pipeline often gets orphaned instead of cleaned up. So you end up with a fresh MV in dap_catalog_dev plus the leftover pipeline still wired to dev_cat, which is exactly the mixed YAML you're seeing (new source dataset, old pipeline name and catalog config).

To get to a clean state:

  1. Drop the materialized view in the new catalog:

DROP MATERIALIZED VIEW dap_catalog_dev.caltrack.vw_item_17_electric_hourly;

  1. Go to the Lakeflow / Delta Live Tables pipelines list (the managed pipelines that back MVs do show up there, they're just not created by hand). Find the orphaned pipeline whose name still contains MV-dev_cat... and delete it manually. That's the piece the drop left behind.

  2. Confirm nothing else references dev_cat. Run DESCRIBE EXTENDED on the recreated objects and check the storage/location line, not just the source dataset, since the hidden materialization tables are where the stale path was hiding.

  3. Recreate the MV fresh in dap_catalog_dev so a brand new backing pipeline is generated against the correct catalog name.

The broader thing worth knowing: renaming a catalog that contains materialized views or streaming tables isn't a clean operation, precisely because of these pinned backing pipelines. You can't rename a catalog in SQL at all (ALTER CATALOG has no RENAME TO), and the Catalog Explorer rename doesn't migrate the internals of MV/ST pipelines. If you have more of these to move, the safer pattern is to treat it as a recreate rather than a rename: create the MVs/STs directly in the target catalog and drop the old ones, rather than renaming the catalog out from under them.