- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-21-2023 04:38 PM
I think it will be usefull if you look at this documentation to understand difrent files and how you can interact with them:
https://learn.microsoft.com/en-us/azure/databricks/files/
there is not much to say then that dbutils is "databricks code" that allows you to work with databricks assests more easily rather than writing your own code.
When you run commands (not spark) in databricks, you basicaly execute it on dirver node, that is linux machine.
Whatever is mounted to this machine, will be avalieble for you from code that can be interpreted in machine.
I know that some of dbutils functions has some performance boost if you want to do some massive mover or remove operation but you can achieve same thing writing your own code from scratch 🙂
if you are very courious what you can find in cluster - create single user cluster, enable web terminal in databricks and launch web terminal. You will be looged as a root to cluster console, where you can explore files.