Need to move files from one Volume to other

[navi_bricks]

We recently enabled Unity catalog on our workspace, as part of certain transformations(Custom clustered Datapipelines(python)) we need to move file from one volume to other volume. 

As the job itself runs on a service principal that has access to external storage, we don't want to pass in any credentials. Can we achieve this? We tried with OS, DbUtils, and Workspace client, all of which need service principal credentials. Finally the reading of volume we achieved through spark context itself, but moving of files we need other way, please help.

[Walter_C]

You should be able to use dbutils.fs.cp to copy the file but you jus need to ensure that the SP has WRITE VOLUME permission on the destination Volume.

[navi_bricks]

Thanks for that,

But I have a Python data pipeline running under a custom cluster, and its not working from there. 

[Walter_C]

What is the error being received? And does the SP has the mentioned permission in UC?

[MujtabaNoori]

Hi @navi_bricks ,
It can be achieved by creating a new notebook and writing the db utils cp or mv command in that notebook. After that, you can create a workflow or an small  independent ADF pipeline using the same SP which has the permission. It will run and move the files.

Thanks

[navi_bricks]

Thanks for that @MujtabaNoori 

Instead of using a notebook, can I use Workspaceclient part of databrick sdk and move the files?

[Walter_C]

You can try:

from databricks.sdk import WorkspaceClient

# Initialize the WorkspaceClient
w = WorkspaceClient()

# Define source and destination paths
source_path = "/Volumes/<source_catalog>/<source_schema>/<source_volume>/<file_name>"
destination_path = "/Volumes/<destination_catalog>/<destination_schema>/<destination_volume>/<file_name>"

# Move the file
w.files.move(source_path, destination_path)

# Verify the file has been moved
for item in w.files.list_directory_contents(f"/Volumes/<destination_catalog>/<destination_schema>/<destination_volume>"):
    print(item.path)

[navi_bricks]

Sorry for the late reply. I was on vacation and didn't check this out. I tried this but always got the error "default auth: cannot configure default credentials." I even tried to use it with client ID and secret being passed as arguments.

[saurabh18cs]

# Using Databricks CLI profile to access the workspace.
w = WorkspaceClient(profile="<<profilename>>")

OR

w = WorkspaceClient(

            host=databricks_host,

            client_id=app_id,

            client_secret=app_secret,

            auth_type="azure-cli",

        )

OR 

from azure.identity import ClientSecretCredential

credential = ClientSecretCredential(

    tenant_id="<tenant-id>",

    client_id="<client-id>",

    client_secret="<client-secret>"

)

# Get the access token for Databricks

token = credential.get_token("https://databricks.azure.net/.default").token

# Initialize the WorkspaceClient using the service principal token

client = WorkspaceClient(

    host="https://<databricks-instance>",

    token=token

)

OR 

# Initialize the WorkspaceClient using an OAuth token

w = WorkspaceClient(

    host="https://<databricks-instance>",

    token="<oauth-token>"

)

OR

w = WorkspaceClient(

        host=databricks_host,

        token=pat_token,

    )

[Dnirmania]

Not all job clusters work well with Volumes. I used following type cluster to access files from Volume.