topic Re: dbutils.fs.mv inefficient with ADLS in Data Engineering

dbutils.fs.mv inefficient with ADLS

ktagseth — Tue, 22 Apr 2025 16:41:38 GMT

dbutils.fs.mv with ADLS currently copies the file and then deletes the old one. This incurs costs and has a lot of overhead vs using the rename functionality in ADLS which is instant and doesn't incur extra costs involved with writing the 'new' data.

Re: dbutils.fs.mv inefficient with ADLS

Louis_Frolio — Tue, 22 Apr 2025 18:55:38 GMT

Is there a question in there or are you sharing you experience with the community?

Re: dbutils.fs.mv inefficient with ADLS

ktagseth — Tue, 22 Apr 2025 19:03:03 GMT

Mostly just sharing my experience, didn't see this documented anywhere and it caused me some headaches.

If i had a question it would be if fs.mv could be updated to use the rename functionality under the hood with adls but that wasn't my intention with the post

Re: dbutils.fs.mv inefficient with ADLS

Louis_Frolio — Tue, 22 Apr 2025 19:16:19 GMT

The tool is really meant for dbfs and is only accessible from within Databricks. If I had to guess the idea is that most folks will not be using dbfs for production or sensitive data (for a host of good reasons) and as such there has not been a big investment in the Databricks Utility. FYI, Unity Catalog volumes are the preferred method for storing all your data and artifacts. Cheers, Lou.