- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2025 06:43 AM
Greetings, total newbie here, please advise if this is an inappropriate forum. I'm trying to learn some basics.
I have tried to copy a file from the local file system onto dbfs, and while it fails in python, it succeeds in the command line. The error in python is puzzling--it appears to complain about some kind of permission.
Given that it works at the command line, I suspect I'm missing some very basic concept, can someone point me at a reference that I can use to fill in the blanks please?
The command issues are illustrated in the image. I have tried to put the text here, but the anti-injection filtering is mangling things, so it's probably broken in ways I haven't noticed:
dbutils.fs.cp("file:/Workspace/Users/214627-ins1@labs.webagesolutions.com/sock_data214627INS1.json", "dbfs:/sock_data214627INS1.json")
The error is:
ExecutionError: (java.lang.SecurityException) Cannot use com.databricks.backend.daemon.driver.WorkspaceLocalFileSystem - local filesystem access is forbidden
And the command-line operations that succeed are (with extras to show that the file wasn't there before the operation 🙂
...$ databricks fs ls -l dbfs: | grep sock_data214
...$ databricks fs cp sock_data214627INS1.json dbfs:
sock_data214627INS1.json -> dbfs:/sock_data214627INS1.json
...$ databricks fs ls -l dbfs: | grep sock_data214
FILE 1528609 2025-07-03T13:24:47Z sock_data214627INS1.json
...$
Thanks for any help you can offer!
Toby
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2025 07:10 AM
Hi @TobyE ,
Try to use compute with Dedicated access mode
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2025 08:21 AM
Hi @szymon_dybczak, thanks for replying. That doesn't appear to have changed anything. But can you give me any clue as to why that might have been expected to change things? The info I saw when creating that compute cluster seemed to indicate that file access would be more restricted, rather than less, so I think there's something more for me to learn there too.
Thanks,
Toby
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2025 08:37 AM
Well, it was indeed a very fundamental misunderstanding. In the UI, I had not connected to my own compute cluster, it was still running "serverless":
So, I guess it's not surprising the process didn't have any privileges, even though the terminal that I spawned off did, sigh 😞
Thanks for trying to help!
Toby