DLT managed by non-existent pipeline

[daz]

I am building out a new DLT pipeline and have since had to rebuild it from scratch. Having deleted the old pipeline and constructed a new one I now get this error:

Table 'X' is already managed by pipeline 'Y'. 

As I only have the one pipeline how would this be possible? It implies deleting the pipeline did not delete all of the related assets or associations. The docs are a little thin on pipeline management.

[daz]

Further to this I can see that there is reference to the old pipeline in the DBFS /pipelines/{Y}/... folder. My instincts tell me that this should have been removed with the interface action of pipeline deletion?

As time is precious I'm going to go ahead and do that with the CLI but not really sure I should/ should have to. It's only a test system after all so meh...

[jose_gonzalez]

Did you publish your DLT tables? docs here https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-quickstart.html#pub...

For more details please check the docs https://docs.databricks.com/data-engineering/delta-live-tables/index.html

[daz]

Hi Jose,

thank you for the reply.

Yes, I did specify a target database. I expect that means they are published.

It's my first attempt at this process and I'm trying to settle on a suitable development approach. Would you suggest keeping everything localised unpublished until it's ready? Ultimately there will be occasions where it is necessary to remove everything and rebuild.

[jose_gonzalez]

Yes, you can keep everything localized. You can just publish the tables other teams needs to access

[Hubert-Dudek]

Please take a look at metastore as there can still be tables.

Deleting a pipeline will not delete data, so except mentioned tables, there can also be delta objects on Delta Lake storage (depends on how it was set)

[Vidula]

Hi there @Darron Smith​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[harvey-c]

Hi, there

I had exactly the same issue with unity catalog as target. Are there any solution to remove the association between table and non-exist pipeline?  Or where are the data of association stored?  I deleted all the files of the schema that contains the table but it was still not working.  

Thank you!

[harvey-c]

If the tables are deleted with catalog explorer,the associated metadata are also deleted. Using SQL to drop the tables may not remove all unity catalog related data including the associated DLT pipeline

[Shinaider777]

rename your function from @Dlt.table, for exemple:

@Dlt.table(
    comment="exemple",
    table_properties={"exemple": "exemple"},
    partition_cols=["a", "b", "c"]
)
def modify_this_name():