topic Upload file from local file system to DBFS (2024) in Data Engineering

Upload file from local file system to DBFS (2024)

Mehdi-LAMRANI — Fri, 24 May 2024 11:02:32 GMT

Recent changes to the worskpace UI (and introduction of Unity Catalog) seem to have discretely sunset the ability to upload data directly to DBFS from the local Filesystem using the UI (NOT the CLI)

I want to be able to load a raw file (no matter the format) and preprocess it through python to be able to *then only* load it into a table or dataframe (it could litteraly be "anything"). All references I can find on the forum and on the internet do not work anymore (Except CLI)

Why such a limitation was "silently" introduced ? I may be blind, but I seem to not to be able to find anything about it in the documentation anymore, it seem to have just vanished into thin air.

Is there still a way to upload a file no matter the format to DBFS ? (or else, I just want to be able to access it programmatically)

Re: Upload file from local file system to DBFS (2024)

mhiltner — Fri, 24 May 2024 17:27:20 GMT

Have you tried using Volumes?

https://docs.databricks.com/en/connect/unity-catalog/volumes.html
You can do it through the UI, on the Catalog Explorer > Add Data button.

Also, you could double check if your workspace admin has disabled DBFS access, because you should still be seeing it on your catalog explorer (a button called "Browse DBFS"). It should be enabled, like this. (Settings > Advanced > Other)

Re: Upload file from local file system to DBFS (2024)

pavithra — Tue, 23 Jul 2024 09:49:59 GMT

not working in community edition