Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I have a table created at unity catalog that was dropped, the files are not deleted due to the 30 day soft delete. Is there anyway to copy the files to a different location? When I try to use dbutils.fs.cp I get location overlap error with unity catalog managed location.
You can use the dbutils.fs.mv command to move the files from the deleted table to a new location. Here's an example of how to do it:
python
# Define the paths source_path = "dbfs:/mnt/<unity-catalog-location>/<database-name>/<table-name>" target_path = "dbfs:/mnt/<new-location>/<path>"
# Move the files using dbutils.fs.mv command dbutils.fs.mv(source_path, target_path)
Replace <unity-catalog-location> with the name of the Unity Catalog location where the table was created, <database-name> with the name of the database containing the table, <table-name> with the name of the dropped table, <new-location> with the new location where you want to move the files, and <path> with any additional path elements needed for the target location.
By using dbutils.fs.mv, you can move the files from the deleted table to a new location without having to copy the files. Please note that moving files from a managed location of Unity Catalog is not recommended and it is against the best practice of using a Catalog. You should only use the Catalog interfaces for performing operations on managed tables.
I hope this helps!
Join Us as a Local Community Builder!
Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!
