dbutils.fs.mv inefficient with ADLS

[ktagseth]

dbutils.fs.mv with ADLS currently copies the file and then deletes the old one. This incurs costs and has a lot of overhead vs using the rename functionality in ADLS which is instant and doesn't incur extra costs involved with writing the 'new' data. 

[BigRoux]

Is there a question in there or are you sharing you experience with the community?

[ktagseth]

Mostly just sharing my experience, didn't see this documented anywhere and it caused me some headaches. 

If i had a question it would be if fs.mv could be updated to use the rename functionality under the hood with adls but that wasn't my intention with the post 

[BigRoux]

The tool is really meant for dbfs and is only accessible from within Databricks.  If I had to guess the idea is that most folks will not be using dbfs for production or sensitive data (for a host of good reasons) and as such there has not been a big investment in the Databricks Utility.  FYI, Unity Catalog volumes are the preferred method for storing all your data and artifacts.  Cheers, Lou.