topic Re: Purge utility for databricks in missing-QuestionPost

Purge utility for databricks

vijaykumar99535 — Thu, 22 Jun 2023 06:08:30 GMT

I have one requirement where I need to delete older data from a folder, and folder data can be partitioned or without partition. What approach to follow to build this utility?

Re: Purge utility for databricks

Anonymous — Fri, 23 Jun 2023 04:34:50 GMT

Hi @vijay kumar

We haven't heard from you since the last response from @Kaniz Fatma , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Re: Purge utility for databricks

vijaykumar99535 — Fri, 23 Jun 2023 05:59:55 GMT

Hi @Vidula Khanna @Kaniz Fatma

It does not answer my real question. We mount adls path in databricks where we read and write the data. Suppose I have one partitioned (partitioned on audit) folder like below. I want to build an utility to delete the 3 days older partition like (/mnt/data/contact/audit=20230620). What is the best approach we can follow to do this. So that it also not impact the parent folder/other partitions.

/mnt/data/contact/audit=20230620

/mnt/data/contact/audit=20230621

/mnt/data/contact/audit=20230622

/mnt/data/contact/audit=20230623