Programatically remove external table completely (Azure)
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2024 02:44 AM
Hi!
I have created external tables with data stored in an Azure storage account.
Is there a way to not only drop the tables but also remove the underlying folder in the storage account which contains the table's data? I want to do this from Databricks in either SQL or Python.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2024 03:40 AM
You can achieve this with the code below. Please review.
table_name = "table_name"
location = "abfss://container@storage-account.dfs.core.windows.net/path/to/table/data/"
spark.sql(f"DROP TABLE IF EXISTS {table_name}")
dbutils.fs.rm(location, recurse=True)
########################################
#Note even you can extract location by doing below
df = spark.sql(f"DESCRIBE DETAIL {table_name}")
location = df.select("location").collect()[0]["location"]
