dbutils.fs.mv inefficient with ADLS
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-22-2025 09:41 AM
dbutils.fs.mv with ADLS currently copies the file and then deletes the old one. This incurs costs and has a lot of overhead vs using the rename functionality in ADLS which is instant and doesn't incur extra costs involved with writing the 'new' data.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-22-2025 11:55 AM
Is there a question in there or are you sharing you experience with the community?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-22-2025 12:03 PM
Mostly just sharing my experience, didn't see this documented anywhere and it caused me some headaches.
If i had a question it would be if fs.mv could be updated to use the rename functionality under the hood with adls but that wasn't my intention with the post
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-22-2025 12:16 PM
The tool is really meant for dbfs and is only accessible from within Databricks. If I had to guess the idea is that most folks will not be using dbfs for production or sensitive data (for a host of good reasons) and as such there has not been a big investment in the Databricks Utility. FYI, Unity Catalog volumes are the preferred method for storing all your data and artifacts. Cheers, Lou.