https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-coalesce-and-repartition-when-it/m-p/22126#M15119 <P>Coalesce essentially groups multiple partitions into a larger partitions. So use coalesce when you want to reduce the number of partitions (and also tasks) without impacting sort order.&nbsp; Ex:- when you want to write-out a single CSV file output instead of multiple parts</P><P></P><P>Use repartition when you want to cause a shuffle that changes the number of partitions.&nbsp; A common use-case for repartition is to remove skew in file sizes or to start out with a smaller/different number of partitions than the default in Spark</P><P></P><P></P> Sun, 20 Jun 2021 03:51:39 GMT aladda 2021-06-20T03:51:39Z https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-coalesce-and-repartition-when-it/m-p/22125#M15118 Sun, 20 Jun 2021 03:49:44 GMT https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-coalesce-and-repartition-when-it/m-p/22125#M15118 aladda 2021-06-20T03:49:44Z https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-coalesce-and-repartition-when-it/m-p/22126#M15119 <P>Coalesce essentially groups multiple partitions into a larger partitions. So use coalesce when you want to reduce the number of partitions (and also tasks) without impacting sort order.&nbsp; Ex:- when you want to write-out a single CSV file output instead of multiple parts</P><P></P><P>Use repartition when you want to cause a shuffle that changes the number of partitions.&nbsp; A common use-case for repartition is to remove skew in file sizes or to start out with a smaller/different number of partitions than the default in Spark</P><P></P><P></P> Sun, 20 Jun 2021 03:51:39 GMT https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-coalesce-and-repartition-when-it/m-p/22126#M15119 aladda 2021-06-20T03:51:39Z