The method in databricks is one that you are using and is slow (repartition(1)).


My blog: https://databrickster.medium.com/