topic Re: removal of Delta live tables in Data Engineering

removal of Delta live tables

Anish_2 — Fri, 21 Feb 2025 08:36:22 GMT

Hello Team,

I have removed definition of table from delta live table pipeline but table is still present in unity catalog. In event log, it is giving below message

Materialized View '`catalog1`.`schema1`.`table1`' is no longer defined in the pipeline and has been marked as inactive in Unity Catalog.
This table will no longer be updated by the pipeline on the next update.
If you want this table to be updated by the pipeline on the next update, you can add the table definition back to the pipeline
which will make the table active again.

Could you please help me why table is marked inactive instead of removal?

Re: removal of Delta live tables

KaranamS — Fri, 21 Feb 2025 19:07:01 GMT

In DLT, when you remove table definition, it is not automatically dropped from Unity Catalog. This is to prevent any accidental data loss and reactivate easily if needed. If you want the table to be deleted from UC, manual cleanup is required.

Re: removal of Delta live tables

Brahmareddy — Fri, 21 Feb 2025 20:52:34 GMT

Hi @Anish_2

How are you doing today? I agree with @KaranamS's answer.

Databricks marks the table as inactive instead of removing it to prevent accidental data loss, allowing you to restore it if needed. Once inactive, the table remains in Unity Catalog but won’t be updated by the pipeline. To fully remove it, you need to manually run DROP TABLE catalog1.schema1.table1; in SQL. If it’s a managed table, use DROP TABLE ... PURGE; to also delete storage files. Databricks doesn’t auto-remove tables when they are removed from the pipeline, so cleanup must be done manually.

Give a try and let me know if this works!

Good day.

Regards,

Brahma

Re: removal of Delta live tables

Anish_2 — Tue, 04 Mar 2025 08:49:10 GMT

We are using on job cluster and i have observed that configuration pipelines.dropInactiveTables false is coming under advanced configuration of cluster. This is coming after latest release of databricks. While creating DLT, i added this property as True then databricks is removing table instead of marking it as inactive. Can you guide me if we can use same property in cluster policy definition?