brickster_2018
Databricks Employee
Databricks Employee

This is by design and working as expected. Spark writes the data distributedly.

use of coalesce (1) can help to generate one file, however this solution is not scalable for large data set as it involves bringing the data to one single task.