Copy files and folder into Users area , but Files are not showing in UI

rkshanmugaraja — Thu, 10 Oct 2024 16:00:12 GMT

Hi All

I'm trying to copy the whole training directory (which contains multiple sub folders and files) from my catalog volume area to each users area.

From : "dbfs:/Volumes/CatalogName/schema/Training"

To : "dbfs:/Workspace/Users/username@domain.com/Training/"

I tried using

1) dbutils.fs.cp(sourcePath , destPath, recursive=true)

Its says it successfully copied but inside the UI its not visible , If I run the ls command , I can see the files and folders.

2) I also tried using the put API /api/2.0/dbfs/put

its the same it says it successful but the files are not visible in the UI.

Is there a way other than UI, because need to copy the folders to 50 users area ? Can't do it manually.

Re: Copy files and folder into Users area , but Files are not showing in UI

radothede — Sun, 13 Oct 2024 10:22:42 GMT

Hi ,

Do You want to use dbfs location on purpose, or You want to upload the training notebooks to Workspace/users location? The reason I'm asking is those are two different locations, although both are related to file management in Databricks. (see: https://docs.databricks.com/en/files/index.html)

If You want to import directory to Workspace/users, one way to go would be using databricks cli.

It is a simple rest api wrapper, so it is easier to handle.

Steps would be to:

Install databricks cli – You can use legacy version using pip install databricks-cli or download latest version using manual provided here: https://docs.databricks.com/en/dev-tools/cli/install.html
Configure the connection settings using databricks configure – easiest way would be to provide host and access token
Use databricks workspace import_dir command to import directory from Your local maschine to Workspace/users locations.

Hope that helps 😊

Best,

Rado.

topic Copy files and folder into Users area , but Files are not showing in UI in Data Engineering

Copy files and folder into Users area , but Files are not showing in UI

Re: Copy files and folder into Users area , but Files are not showing in UI