Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I have a table created at unity catalog that was dropped, the files are not deleted due to the 30 day soft delete. Is there anyway to copy the files to a different location? When I try to use dbutils.fs.cp I get location overlap error with unity catalog managed location.
You can use the dbutils.fs.mv command to move the files from the deleted table to a new location. Here's an example of how to do it:
python
# Define the paths source_path = "dbfs:/mnt/<unity-catalog-location>/<database-name>/<table-name>" target_path = "dbfs:/mnt/<new-location>/<path>"
# Move the files using dbutils.fs.mv command dbutils.fs.mv(source_path, target_path)
Replace <unity-catalog-location> with the name of the Unity Catalog location where the table was created, <database-name> with the name of the database containing the table, <table-name> with the name of the dropped table, <new-location> with the new location where you want to move the files, and <path> with any additional path elements needed for the target location.
By using dbutils.fs.mv, you can move the files from the deleted table to a new location without having to copy the files. Please note that moving files from a managed location of Unity Catalog is not recommended and it is against the best practice of using a Catalog. You should only use the Catalog interfaces for performing operations on managed tables.
I hope this helps!
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.