topic Re: Need to move files from one Volume to other in Administration & Architecture

Need to move files from one Volume to other

navi_bricks — Thu, 19 Dec 2024 08:41:09 GMT

We recently enabled Unity catalog on our workspace, as part of certain transformations(Custom clustered Datapipelines(python)) we need to move file from one volume to other volume.

As the job itself runs on a service principal that has access to external storage, we don't want to pass in any credentials. Can we achieve this? We tried with OS, DbUtils, and Workspace client, all of which need service principal credentials. Finally the reading of volume we achieved through spark context itself, but moving of files we need other way, please help.

Re: Need to move files from one Volume to other

Walter_C — Thu, 19 Dec 2024 11:40:30 GMT

You should be able to use dbutils.fs.cp to copy the file but you jus need to ensure that the SP has WRITE VOLUME permission on the destination Volume.

Re: Need to move files from one Volume to other

navi_bricks — Thu, 19 Dec 2024 11:47:35 GMT

Thanks for that,

But I have a Python data pipeline running under a custom cluster, and its not working from there.

Re: Need to move files from one Volume to other

Walter_C — Thu, 19 Dec 2024 11:51:07 GMT

What is the error being received? And does the SP has the mentioned permission in UC?

Re: Need to move files from one Volume to other

MujtabaNoori — Mon, 23 Dec 2024 07:24:16 GMT

Hi @navi_bricks ,
It can be achieved by creating a new notebook and writing the db utils cp or mv command in that notebook. After that, you can create a workflow or an small independent ADF pipeline using the same SP which has the permission. It will run and move the files.

Thanks

Re: Need to move files from one Volume to other

navi_bricks — Tue, 24 Dec 2024 13:30:16 GMT

Thanks for that @MujtabaNoori

Instead of using a notebook, can I use Workspaceclient part of databrick sdk and move the files?

Re: Need to move files from one Volume to other

Walter_C — Fri, 03 Jan 2025 22:45:31 GMT

You can try:

from databricks.sdk import WorkspaceClient # Initialize the WorkspaceClient w = WorkspaceClient() # Define source and destination paths source_path = "/Volumes/<source_catalog>/<source_schema>/<source_volume>/<file_name>" destination_path = "/Volumes/<destination_catalog>/<destination_schema>/<destination_volume>/<file_name>" # Move the file w.files.move(source_path, destination_path) # Verify the file has been moved for item in w.files.list_directory_contents(f"/Volumes/<destination_catalog>/<destination_schema>/<destination_volume>"): print(item.path)

Re: Need to move files from one Volume to other

navi_bricks — Wed, 15 Jan 2025 13:02:20 GMT

Sorry for the late reply. I was on vacation and didn't check this out. I tried this but always got the error "default auth: cannot configure default credentials." I even tried to use it with client ID and secret being passed as arguments.

Re: Need to move files from one Volume to other

saurabh18cs — Wed, 15 Jan 2025 15:45:48 GMT

# Using Databricks CLI profile to access the workspace.
w = WorkspaceClient(profile="<<profilename>>")

w = WorkspaceClient(

host=databricks_host,

client_id=app_id,

client_secret=app_secret,

auth_type="azure-cli",

)

from azure.identity import ClientSecretCredential

credential = ClientSecretCredential(

tenant_id="<tenant-id>",

client_id="<client-id>",

client_secret="<client-secret>"

)

# Get the access token for Databricks

token = credential.get_token("https://databricks.azure.net/.default").token

# Initialize the WorkspaceClient using the service principal token

client = WorkspaceClient(

host="https://<databricks-instance>",

token=token

)

# Initialize the WorkspaceClient using an OAuth token

w = WorkspaceClient(

host="https://<databricks-instance>",

token="<oauth-token>"

)

w = WorkspaceClient(

host=databricks_host,

token=pat_token,

)

Re: Need to move files from one Volume to other

Dnirmania — Wed, 15 Jan 2025 16:45:22 GMT

Not all job clusters work well with Volumes. I used following type cluster to access files from Volume.