topic Re: How do I drop a delta live table? in Data Engineering

How do I drop a delta live table?

TamD — Thu, 10 Oct 2024 19:10:47 GMT

I'm a newbie and I've just done the "Run your first Delta Live Tables pipeline" tutorial.

The tutorial downloads a publicly available csv baby names file and creates two new Delta Live tables from it. Now I want to be a good dev and clean up the resources so they aren't cluttering up the environment. However, I can't find docco for dropping delta live tables. You can't just drop them separately, because you get an error:

[STREAMING_TABLE_OPERATION_NOT_ALLOWED.DROP_DELTA_LIVE_TABLE] The operation DROP is not allowed: The operation does not apply to Streaming Tables created from Delta Live Tables, instead remove the Streaming Table from the pipeline definition in Delta Live Tables.

I removed the definitions of the two tables from the pipeline definition (per the above error), but then I get the error:

[DLT ERROR CODE: NO_TABLES_IN_PIPELINE] You attempted to update an empty pipeline. This error usually means no tables were discovered in your specified source code. Please verify that your source code includes table definitions.

What is the correct way to remove unwanted Delta Live Tables?

Re: How do I drop a delta live table?

filipniziol — Thu, 10 Oct 2024 20:48:15 GMT

Hi @TamD ,

1. Delete the DLT Pipeline
2. Delete the delta tables created by DLT Pipeline

Re: How do I drop a delta live table?

TamD — Thu, 10 Oct 2024 22:01:28 GMT

Hi @filipniziol thank you for replying. Have you confirmed that your method works? And if you have, could you please provide more detail for the "delete the delta tables" part? As I describe above, attempting to drop the tables outside of the pipeline results in an error.

Re: How do I drop a delta live table?

gchandra — Fri, 11 Oct 2024 11:17:38 GMT

As DLT is a declarative coding, the pipeline is tightly coupled with the DLT table.

Deleting the pipeline or removing that particular table from the pipeline and rerunning it will remove the DLT table.

Re: How do I drop a delta live table?

TamD — Sun, 13 Oct 2024 20:49:38 GMT

Thank you @gchandra . Deleting the pipeline does indeed remove the materialized view definitions from the Catalog. How can I confirm that the underlying S3 storage has also been cleared? Just removing the pointers in the Catalog is not enough, if it leaves orphaned resources in cloud storage.

Re: How do I drop a delta live table?

MartinIsti — Mon, 14 Oct 2024 23:29:09 GMT

Deleting the DLT pipeline should delete the underlying data files, too. But as DLT tables / materialized views don't show us their ID or storage location, it is not trivial to find them and to make sure they are actually removed from storage right away or there is an intentional latency (e.g. 7 days as for delta tables).

Re: How do I drop a delta live table?

celsochokiti — Mon, 28 Oct 2024 16:35:30 GMT

Hi, @filipniziol

One help, I have a doubt, how can i delete a delta tables created by DLT Pipeline if i haven't more acess to pipeline, the workspace is unaccessible.

Re: How do I drop a delta live table?

ImranA — Mon, 09 Dec 2024 11:00:42 GMT

@gchandra for example a table called "cars", if I remove the table from DLT pipeline and drop the table from catalog. Now if I change the schema of the table, and create the table again using the same table name "cars" through the same pipeline, Why I am getting schema mismatched error?

Why it didn't remove the schema, is there any other files that we need to be aware of in the backend which doesnt get deleted automatically?