topic Re: DLT managed by non-existent pipeline in Data Engineering

DLT managed by non-existent pipeline

daz — Tue, 26 Jul 2022 23:30:02 GMT

I am building out a new DLT pipeline and have since had to rebuild it from scratch. Having deleted the old pipeline and constructed a new one I now get this error:

Table 'X' is already managed by pipeline 'Y'.

As I only have the one pipeline how would this be possible? It implies deleting the pipeline did not delete all of the related assets or associations. The docs are a little thin on pipeline management.

Re: DLT managed by non-existent pipeline

daz — Tue, 26 Jul 2022 23:39:37 GMT

Further to this I can see that there is reference to the old pipeline in the DBFS /pipelines/{Y}/... folder. My instincts tell me that this should have been removed with the interface action of pipeline deletion?

As time is precious I'm going to go ahead and do that with the CLI but not really sure I should/ should have to. It's only a test system after all so meh...

Re: DLT managed by non-existent pipeline

Hubert-Dudek — Wed, 27 Jul 2022 10:46:56 GMT

Please take a look at metastore as there can still be tables.

Deleting a pipeline will not delete data, so except mentioned tables, there can also be delta objects on Delta Lake storage (depends on how it was set)

Re: DLT managed by non-existent pipeline

jose_gonzalez — Fri, 29 Jul 2022 23:34:59 GMT

Did you publish your DLT tables? docs here https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-quickstart.html#publish-datasets

For more details please check the docs https://docs.databricks.com/data-engineering/delta-live-tables/index.html

Re: DLT managed by non-existent pipeline

daz — Sun, 31 Jul 2022 22:45:35 GMT

Hi Jose,

thank you for the reply.

Yes, I did specify a target database. I expect that means they are published.

It's my first attempt at this process and I'm trying to settle on a suitable development approach. Would you suggest keeping everything localised unpublished until it's ready? Ultimately there will be occasions where it is necessary to remove everything and rebuild.

Re: DLT managed by non-existent pipeline

jose_gonzalez — Wed, 17 Aug 2022 20:59:32 GMT

Yes, you can keep everything localized. You can just publish the tables other teams needs to access

Re: DLT managed by non-existent pipeline

Vidula — Tue, 06 Sep 2022 09:09:36 GMT

Hi there @Darron Smith

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: DLT managed by non-existent pipeline

harvey-c — Fri, 23 Feb 2024 02:05:45 GMT

Hi, there

I had exactly the same issue with unity catalog as target. Are there any solution to remove the association between table and non-exist pipeline? Or where are the data of association stored? I deleted all the files of the schema that contains the table but it was still not working.

Thank you!

Re: DLT managed by non-existent pipeline

harvey-c — Sun, 10 Mar 2024 10:15:54 GMT

If the tables are deleted with catalog explorer,the associated metadata are also deleted. Using SQL to drop the tables may not remove all unity catalog related data including the associated DLT pipeline

Re: DLT managed by non-existent pipeline

Shinaider777 — Wed, 01 May 2024 17:34:38 GMT

rename your function from @Dlt.table, for exemple:

@Dlt.table(
comment="exemple",
table_properties={"exemple": "exemple"},
partition_cols=["a", "b", "c"]
)
def modify_this_name():