@David Toft​ Hi, The current implementation of dbutils.fs is single-threaded, performs the initial listing on the driver and subsequently launches a Spark job to perform the per-file operations. So I guess the put operation is running on a single core and could eventually break.

Did you try the possibility of storing the variable as text dataframe and writing to a path using dataframe writer ?