topic Databricks drop and remove s3 storage files safely in Data Engineering

Databricks drop and remove s3 storage files safely

Abela — Mon, 22 Nov 2021 17:26:11 GMT

After dropping a delta table using DROP command in databricks, is there a way to drop the s3 files in databricks without using rm command? Looking for a solution where junior developers can safely drop a table wihout messing with the rm command where they may cause accidental data loss using recursive option.

thanks

Alina.

Re: Databricks drop and remove s3 storage files safely

Hubert-Dudek — Mon, 22 Nov 2021 18:41:15 GMT

official way is that before DROP:

Run DELETE FROM:

DELETE FROM events

Run VACUUM with an interval of zero:

VACUUM events RETAIN 0 HOURS

I agree that there could be some DEEP DROP 🙂

Alternatively not in SQL but in python you could write custom class/function to do that and then preinstall it on clusters so people would use some CleanTable(TableName) to make data validation and then delete+vacuum+drop+rm

Re: Databricks drop and remove s3 storage files safely

jose_gonzalez — Mon, 22 Nov 2021 19:51:12 GMT

Hi @Alina Bella ,

Like @Hubert Dudek mentioned, we have a best practice guide for dropping managed tables. You can find the docs here

Re: Databricks drop and remove s3 storage files safely

jose_gonzalez — Mon, 29 Nov 2021 19:14:17 GMT

Hi @Alina Bella ,

If @Hubert Dudek ''s answer solved the issue, would you be happy to mark their answer as best? That will help others find the solution more easily in the future.