topic Re: Problem using dbutils to access local files in Data Engineering

Problem using dbutils to access local files

TobyE — Thu, 03 Jul 2025 13:43:56 GMT

Greetings, total newbie here, please advise if this is an inappropriate forum. I'm trying to learn some basics.

I have tried to copy a file from the local file system onto dbfs, and while it fails in python, it succeeds in the command line. The error in python is puzzling--it appears to complain about some kind of permission.

Given that it works at the command line, I suspect I'm missing some very basic concept, can someone point me at a reference that I can use to fill in the blanks please?

The command issues are illustrated in the image. I have tried to put the text here, but the anti-injection filtering is mangling things, so it's probably broken in ways I haven't noticed:

dbutils.fs.cp("file:/Workspace/Users/214627-ins1@labs.webagesolutions.com/sock_data214627INS1.json", "dbfs:/sock_data214627INS1.json")

The error is:

ExecutionError: (java.lang.SecurityException) Cannot use com.databricks.backend.daemon.driver.WorkspaceLocalFileSystem - local filesystem access is forbidden

And the command-line operations that succeed are (with extras to show that the file wasn't there before the operation 🙂

...$ databricks fs ls -l dbfs: | grep sock_data214
...$ databricks fs cp sock_data214627INS1.json dbfs:
sock_data214627INS1.json -> dbfs:/sock_data214627INS1.json
...$ databricks fs ls -l dbfs: | grep sock_data214
FILE 1528609 2025-07-03T13:24:47Z sock_data214627INS1.json
...$

Thanks for any help you can offer!

Toby

Re: Problem using dbutils to access local files

szymon_dybczak — Thu, 03 Jul 2025 14:10:50 GMT

Hi @TobyE ,

Try to use compute with Dedicated access mode

Re: Problem using dbutils to access local files

TobyE — Thu, 03 Jul 2025 15:21:18 GMT

Hi @szymon_dybczak, thanks for replying. That doesn't appear to have changed anything. But can you give me any clue as to why that might have been expected to change things? The info I saw when creating that compute cluster seemed to indicate that file access would be more restricted, rather than less, so I think there's something more for me to learn there too.

Thanks,
Toby

Re: Problem using dbutils to access local files

TobyE — Thu, 03 Jul 2025 15:37:41 GMT

Well, it was indeed a very fundamental misunderstanding. In the UI, I had not connected to my own compute cluster, it was still running "serverless":

So, I guess it's not surprising the process didn't have any privileges, even though the terminal that I spawned off did, sigh 😞

Thanks for trying to help!
Toby