topic Re: Where / how does DBFS store files? in Data Engineering

Where / how does DBFS store files?

al_joe — Sat, 05 Feb 2022 08:19:12 GMT

I tried to use %fs head to print the contents of a CSV file used in a training

%fs head "/mnt/path/file.csv"

but got an error saying cannot head a directory!?

Then I did %fs ls on the same CSV file and got a list of 4 files under a directory named as a CSV file.

A little confused how DBFS stores files on underlying storage and where they are stored. I am using community edition.

any pointers appreciated. Thanks.

EDIT: Apache iceberg has a illustration that shows the internal logical/storage structure well. Is there something similar for DBFS?

Re: Where / how does DBFS store files?

Chris_Shehu — Mon, 07 Feb 2022 15:47:23 GMT

Is this a fresh cluster startup? I noticed that if you browse tot he location using the UI you get a prompt asking you to attach a cluster on the first startup. I would make sure that's setup and make sure you can see it in the directory.

Also have you tried loading the csv into pandas to display it?

Re: Where / how does DBFS store files?

User16753725182 — Wed, 02 Mar 2022 14:56:09 GMT

Hi @Al Jo , are you still seeing the error while printing the contents of te CSV file?