https://community.databricks.com/t5/data-engineering/how-to-write-only-file-on-to-the-blob-or-adls-from-databricks/m-p/57605#M30841 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/98328">@Simha</a>&nbsp;, This is expected behavior. Spark always creates an output directory when writing the data and it divides the result into multiple part files. This is because multiple executors write the result into the output directory. We cannot make the spark write the file without creating the output directory.</P> <P>But we can control the no. of part files that are written in output directory by using the coalesce function. To get a single file output, you can use coalesce(1) while doing the write operation. However, I would advise you to decide the coalesce partition carefully as coalesce(1) would bring all the data to single executor and if the data volume is huge, this can lead to executor going OOM.&nbsp;</P> Wed, 17 Jan 2024 15:49:03 GMT Lakshay 2024-01-17T15:49:03Z https://community.databricks.com/t5/data-engineering/how-to-write-only-file-on-to-the-blob-or-adls-from-databricks/m-p/57582#M30834 <P>Hi All,</P><P>I am trying to write a csv file on to the blob and ADLS from databricks notebook using pyspark and a separate folder is created with the mentioned filename and a partition is created within the folder.</P><P>I want only file to be written.&nbsp;</P><P>Can anyone help me to fix this issue.</P><P>Thanks in advance.</P> Wed, 17 Jan 2024 12:43:42 GMT https://community.databricks.com/t5/data-engineering/how-to-write-only-file-on-to-the-blob-or-adls-from-databricks/m-p/57582#M30834 Simha 2024-01-17T12:43:42Z https://community.databricks.com/t5/data-engineering/how-to-write-only-file-on-to-the-blob-or-adls-from-databricks/m-p/57605#M30841 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/98328">@Simha</a>&nbsp;, This is expected behavior. Spark always creates an output directory when writing the data and it divides the result into multiple part files. This is because multiple executors write the result into the output directory. We cannot make the spark write the file without creating the output directory.</P> <P>But we can control the no. of part files that are written in output directory by using the coalesce function. To get a single file output, you can use coalesce(1) while doing the write operation. However, I would advise you to decide the coalesce partition carefully as coalesce(1) would bring all the data to single executor and if the data volume is huge, this can lead to executor going OOM.&nbsp;</P> Wed, 17 Jan 2024 15:49:03 GMT https://community.databricks.com/t5/data-engineering/how-to-write-only-file-on-to-the-blob-or-adls-from-databricks/m-p/57605#M30841 Lakshay 2024-01-17T15:49:03Z