topic Any way to access unity catalog location through python/dbutils in Data Engineering

Any way to access unity catalog location through python/dbutils

ashish577 — Thu, 03 Aug 2023 05:49:29 GMT

I have a table created at unity catalog that was dropped, the files are not deleted due to the 30 day soft delete. Is there anyway to copy the files to a different location? When I try to use dbutils.fs.cp I get location overlap error with unity catalog managed location.

Re: Any way to access unity catalog location through python/dbutils

Atanu — Thu, 03 Aug 2023 16:57:58 GMT

I think you can not do that, as this are already into soft delete.

what is the actual error you got?

Re: Any way to access unity catalog location through python/dbutils

Tharun-Kumar — Thu, 03 Aug 2023 18:06:50 GMT

@ashish577

You can restore the file using your cloud service provider. Then you can create a table on the top of the restored files.

Re: Any way to access unity catalog location through python/dbutils

youssefmrini — Tue, 08 Aug 2023 15:56:26 GMT

You can use the dbutils.fs.mv command to move the files from the deleted table to a new location. Here's an example of how to do it:

python

# Define the paths
source_path = "dbfs:/mnt/<unity-catalog-location>/<database-name>/<table-name>"
target_path = "dbfs:/mnt/<new-location>/<path>"

# Move the files using dbutils.fs.mv command
dbutils.fs.mv(source_path, target_path)

Replace <unity-catalog-location> with the name of the Unity Catalog location where the table was created, <database-name> with the name of the database containing the table, <table-name> with the name of the dropped table, <new-location> with the new location where you want to move the files, and <path> with any additional path elements needed for the target location.

By using dbutils.fs.mv, you can move the files from the deleted table to a new location without having to copy the files. Please note that moving files from a managed location of Unity Catalog is not recommended and it is against the best practice of using a Catalog. You should only use the Catalog interfaces for performing operations on managed tables.

I hope this helps!