โ09-13-2022 05:47 PM
the documentation states that "drop table":
Deletes the table and removes the directory associated with the table from the file system if the table is not
EXTERNAL table. An exception is thrown if the table does not exist.
In case of an external table, only the associated metadata information is removed from the metastore schema.
This does not work!!
I have a managed table, see below, managed and stored on a mounted Azure storage account:
then I execute spark.sql("drop table silver.company")
the result:
โ07-28-2023 05:13 AM
Are you using Unity Catalog? If yes, then the data will be deleted from cloud tenant in next 30 days.
https://docs.databricks.com/data-governance/unity-catalog/create-tables.html#managed-tables
โ09-14-2022 09:23 PM
Hi, Could you please mention the document which was followed for the issue and also to configure the external metadata? Also, could you please state the steps.
โ09-15-2022 10:45 AM
to mount a drive in databricks via the following python code
mountPoint = "/mnt/bronze"
accountName = "<provide account name>"
containerName = "bronze"
account_key = "<provide account key>"
dbutils.fs.mount(
source = f"wasbs://{containerName}@{accountName}.blob.core.windows.net",
mount_point = mountPoint,
extra_configs = {f"fs.azure.account.key.{accountName}.blob.core.windows.net" : account_key} )
to create a table just CREATE ...
then drop, the directory with all files remain after dropping a table
โ09-27-2022 05:16 AM
Hi @Maciej Gโ
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!
โ07-28-2023 03:20 AM
Hi, did you solve this issue? I have the very same issue with managed tables now.
โ07-28-2023 05:13 AM
Are you using Unity Catalog? If yes, then the data will be deleted from cloud tenant in next 30 days.
https://docs.databricks.com/data-governance/unity-catalog/create-tables.html#managed-tables
โ07-28-2023 08:55 AM
@alesventus how you are creating your table, is that in unity catalog and are you using managed with external location. usually you should not specify any location for managed table . by default it gets created in hive metastore if you are not using unity catalog. please send what type of query you have used to create managed table.
if location is not used during table creation, then drop should work
โ07-30-2023 11:54 PM
Hi @karthik_p , correct answer is the one given by @Lakshay There is 30 days retention after delete of managed table.
โ09-14-2023 01:21 AM
Hi,
There is a way to force delete files after drop the table and don't wait 30 days to see size in S3 decrease?
Tables that I dropped related to the dev and staging, I don't want to keep there files for 30 days
a week ago
Perhaps this post might be useful to you:
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group