topic Deleted schema leads to DLT pipeline problems in Data Engineering

Deleted schema leads to DLT pipeline problems

jorhona — Tue, 27 May 2025 10:43:05 GMT

Hello.

When testing a dlt table pipeline i accidentally mispelt the target schema. The pipeline worked and created the tables. After realising my mistake, i deleted the tables and the schema - thinking nothing of it.

However when I run the pipeline with the corrected schema, I get the following error :

"""

Soft-deleted MV/STs that require changes cannot be undropped directly. If you need to update the target schema of the pipeline or modify the visibility of an MV/ST while also undropping it, please invoke the undrop operation with the original schema and visibility in an update first, before applying the changes in a subsequent update.

The following soft-deleted MV/STs required changes: table1, table2 etc etc

"""

I cant get back the tables to drop them properly.

What do i do ?

Thanks 🙂

Re: Deleted schema leads to DLT pipeline problems

SP_6721 — Tue, 27 May 2025 14:08:34 GMT

Hi @jorhona ,

This error happens because DLT marks deleted tables as "soft-deleted" in Unity Catalog, and they need to be restored to their original state before you can make changes.
To resolve this:

Recover the misspelled schema and tables:
UNDROP SCHEMA catalog_name.<typo_schema>;
UNDROP TABLE catalog_name.<typo_schema>.table1;
Then drop them properly:
DROP TABLE catalog_name.<typo_schema>.table1;
DROP SCHEMA catalog_name.<typo_schema>;

Once that's done, you can rerun your pipeline using the corrected schema without hitting the soft-deletion issue.

Re: Deleted schema leads to DLT pipeline problems

jorhona — Wed, 28 May 2025 11:06:42 GMT

In the end i deleted and recreated the pipeline which fixed the problem.

Luckily it was only in dev so didnt lose any history of pipeline success etc in prod.

Still, is a bit of a pain for dlt, along with the problem of multiple developers not being able to work on a single pipeline "this table belongs to another pipeline".