topic Efficiently move multiple files with dbutils.fs.mv command on abfs storage in Data Engineering

Efficiently move multiple files with dbutils.fs.mv command on abfs storage

Dean_Lovelace — Wed, 17 May 2023 08:36:23 GMT

As part of my batch processing I archive a large number of small files received from the source system each day using the dbutils.fs.mv command.

This takes hours as dbutils.fs.mv moves the files one at a time.

How can I speed this up?

daniel_sahal — Wed, 17 May 2023 09:02:06 GMT

@Dean Lovelace

You can use multithreading.