07-26-2022 04:30 PM
I am building out a new DLT pipeline and have since had to rebuild it from scratch. Having deleted the old pipeline and constructed a new one I now get this error:
Table 'X' is already managed by pipeline 'Y'.
As I only have the one pipeline how would this be possible? It implies deleting the pipeline did not delete all of the related assets or associations. The docs are a little thin on pipeline management.
07-26-2022 04:39 PM
Further to this I can see that there is reference to the old pipeline in the DBFS /pipelines/{Y}/... folder. My instincts tell me that this should have been removed with the interface action of pipeline deletion?
As time is precious I'm going to go ahead and do that with the CLI but not really sure I should/ should have to. It's only a test system after all so meh...
07-29-2022 04:34 PM
Did you publish your DLT tables? docs here https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-quickstart.html#pub...
For more details please check the docs https://docs.databricks.com/data-engineering/delta-live-tables/index.html
07-31-2022 03:45 PM
Hi Jose,
thank you for the reply.
Yes, I did specify a target database. I expect that means they are published.
It's my first attempt at this process and I'm trying to settle on a suitable development approach. Would you suggest keeping everything localised unpublished until it's ready? Ultimately there will be occasions where it is necessary to remove everything and rebuild.
08-17-2022 01:59 PM
Yes, you can keep everything localized. You can just publish the tables other teams needs to access
07-27-2022 03:46 AM
Please take a look at metastore as there can still be tables.
Deleting a pipeline will not delete data, so except mentioned tables, there can also be delta objects on Delta Lake storage (depends on how it was set)
09-06-2022 02:09 AM
Hi there @Darron Smith
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!
02-22-2024 06:05 PM
Hi, there
I had exactly the same issue with unity catalog as target. Are there any solution to remove the association between table and non-exist pipeline? Or where are the data of association stored? I deleted all the files of the schema that contains the table but it was still not working.
Thank you!
03-10-2024 03:15 AM
If the tables are deleted with catalog explorer,the associated metadata are also deleted. Using SQL to drop the tables may not remove all unity catalog related data including the associated DLT pipeline
05-01-2024 10:34 AM
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group