You can use the below command to delete the particular file
dbutils.fs.rm("path of the file")
If you want to delete the entire directory where it consists of sub-directories and files, you can use the below command to delete the files recursively
dbutils.fs.rm("path of the folder",True)
After executing the above commands you will be getting the output below, to confirm file/directory got deleted successfully.
You can use the following method to delete only the duplicate tables
%scala
val tables = spark.sql("""SHOW TABLES""").createOrReplaceTempView("tables")
val temp_tables = spark.sql("""select tableName from tables where tableName like '%-1%' """)
temp_tables.collect().foreach(row => println("DROP TABLE IF EXISTS " + row.toString().replace("[", "").replace("]", "") + ";"))
You will get the sql command in a string...you can either copy this cell output to another cell directly and run it or you can automate the process by storing this output into a variable and then calling it through a loop...Better way would be to simply copy the output and execute it.
Hi @Mahesh Babu Uppala (Customer) , We haven’t heard from you on the last response from @Uma Maheswara Rao Desula and @Ratna Chaitanya Raju Bandaru , and I was checking back to see if their suggestions helped you.
Or else, If you have any solution, please do share that with the community as it can be helpful to others.
Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
Welcome to Databricks Community: Lets learn, network and celebrate together
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
