topic Re: dropping a managed table does not remove the underlying files in Data Engineering

dropping a managed table does not remove the underlying files

my_community2 — Wed, 14 Sep 2022 00:47:13 GMT

the documentation states that "drop table":

Deletes the table and removes the directory associated with the table from the file system if the table is not

EXTERNAL table. An exception is thrown if the table does not exist.

In case of an external table, only the associated metadata information is removed from the metastore schema.

This does not work!!

I have a managed table, see below, managed and stored on a mounted Azure storage account:

then I execute spark.sql("drop table silver.company")

the result:

metadata is deleted from the catalog
folder is NOT deleted from disk (it should have been deleted as per docs)

Re: dropping a managed table does not remove the underlying files

my_community2 — Thu, 15 Sep 2022 17:45:46 GMT

docs at https://docs.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/sql-ref-syntax-ddl-drop-table

to mount a drive in databricks via the following python code

mountPoint = "/mnt/bronze"

accountName = "<provide account name>"

containerName = "bronze"

account_key = "<provide account key>"

dbutils.fs.mount(

source = f"wasbs://{containerName}@{accountName}.blob.core.windows.net",

mount_point = mountPoint,

extra_configs = {f"fs.azure.account.key.{accountName}.blob.core.windows.net" : account_key} )

to create a table just CREATE ...

then drop, the directory with all files remain after dropping a table

Re: dropping a managed table does not remove the underlying files

Anonymous — Tue, 27 Sep 2022 12:16:57 GMT

Hi @Maciej G

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: dropping a managed table does not remove the underlying files

Debayan — Thu, 15 Sep 2022 04:23:33 GMT

Hi, Could you please mention the document which was followed for the issue and also to configure the external metadata? Also, could you please state the steps.

Re: dropping a managed table does not remove the underlying files

alesventus — Fri, 28 Jul 2023 10:20:05 GMT

Hi, did you solve this issue? I have the very same issue with managed tables now.

Re: dropping a managed table does not remove the underlying files

Lakshay — Fri, 28 Jul 2023 12:13:17 GMT

Are you using Unity Catalog? If yes, then the data will be deleted from cloud tenant in next 30 days.

https://docs.databricks.com/data-governance/unity-catalog/create-tables.html#managed-tables

Re: dropping a managed table does not remove the underlying files

karthik_p — Fri, 28 Jul 2023 15:55:53 GMT

@alesventus how you are creating your table, is that in unity catalog and are you using managed with external location. usually you should not specify any location for managed table . by default it gets created in hive metastore if you are not using unity catalog. please send what type of query you have used to create managed table.

if location is not used during table creation, then drop should work

Re: dropping a managed table does not remove the underlying files

alesventus — Mon, 31 Jul 2023 06:54:13 GMT

Hi @karthik_p , correct answer is the one given by @Lakshay There is 30 days retention after delete of managed table.

Re: dropping a managed table does not remove the underlying files

MajdSAAD_7953 — Thu, 14 Sep 2023 08:21:18 GMT

Hi,

There is a way to force delete files after drop the table and don't wait 30 days to see size in S3 decrease?

Tables that I dropped related to the dev and staging, I don't want to keep there files for 30 days

Re: dropping a managed table does not remove the underlying files

n_joy — Mon, 30 Sep 2024 14:12:50 GMT

Perhaps this post might be useful to you:

https://community.databricks.com/t5/data-engineering/databricks-drop-and-remove-s3-storage-files-safely/m-p/34569#M25311