topic Re: dbutils.fs.ls versus pathlib.Path in Get Started Discussions

dbutils.fs.ls versus pathlib.Path

chari — Thu, 21 Dec 2023 17:25:57 GMT

Hello community members,

The dbutils.fs.ls('/') exposes the distributed file system(DBFS) on the databricks cluster. Similary, the python library pathlib can also expose 4 files in the cluster like below:

from pathlib import Path

mypath = Path('/')

for item, num in mypath.iterdir():

print(num, item)

How do they differ and what kind of files do they allow to work with ?

Thanks

Re: dbutils.fs.ls versus pathlib.Path

Wojciech_BUK — Fri, 22 Dec 2023 00:38:29 GMT

I think it will be usefull if you look at this documentation to understand difrent files and how you can interact with them:
https://learn.microsoft.com/en-us/azure/databricks/files/

there is not much to say then that dbutils is "databricks code" that allows you to work with databricks assests more easily rather than writing your own code.

When you run commands (not spark) in databricks, you basicaly execute it on dirver node, that is linux machine.
Whatever is mounted to this machine, will be avalieble for you from code that can be interpreted in machine.

I know that some of dbutils functions has some performance boost if you want to do some massive mover or remove operation but you can achieve same thing writing your own code from scratch 🙂

if you are very courious what you can find in cluster - create single user cluster, enable web terminal in databricks and launch web terminal. You will be looged as a root to cluster console, where you can explore files.

Re: dbutils.fs.ls versus pathlib.Path

chari — Sun, 24 Dec 2023 09:48:58 GMT

Hello Wojceich,

Thanks for your time but I like to correct myself a bit. The DBFS (databricks file system) is actually Azure blob storage which is mounted on databricks whereas the libraries are installed on the driver node (as you rightly pointed out!).