08-31-2023 10:30 AM - edited 08-31-2023 10:33 AM
I've been struggling with figuring out how to delete a managed delta live table. If I run a drop command in Databricks SQL I get:
[STREAMING_TABLE_OPERATION_NOT_ALLOWED.DROP_DELTA_LIVE_TABLE] The operation DROP is not allowed: The operation does not apply to Streaming Tables created from Delta Live Tables, instead remove the Streaming Table from the pipeline definition in Delta Live Tables.
If I go to the Data section and click the 3 dots and Delete, it only deletes it from the UI. The underlying data is all there. A subsequent run will show all the old data again.
The only solution I've been able to do is to create a new DLT pipeline, and a new target schema, and then re-point my DLT process to them. And then run. There must be a better way. What do I do to be able to completely remove the underlying data of a DLT table?
09-06-2023 03:32 AM
Hi @dbdude, To completely remove the underlying data of a Delta Live Table (DLT), you need to manually delete the data stored in the path. When you drop a table, only the metadata gets dropped and the underlying data remains untouched. The DROP TABLE command doesn't apply to Streaming Tables created from Delta Live Tables. Instead, you need to remove the Streaming Table from the pipeline definition in Delta Live Tables.
10-03-2023 12:18 AM
@Kaniz_Fatma - Thanks - how do i track their right guid in order to delete the correct folder?
10-10-2023 04:13 AM - edited 10-10-2023 04:21 AM
Hi @dbdude , To track the right GUID (Globally Unique Identifier) for folder deletion, follow the audit logs.
Monday
@Kaniz_Fatma , How to remove Streaming Table from the pipeline definition in Delta Live Tables?
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!