Compression Export to volume is not working as expected

[rakshakpr11]

I am trying to write data into a volume using below 

table.coalesce(1)

          .write

          .mode("overwrite")

          .format(file_format)

          .option("header", "true")

          .option("delimiter", field_delimiter)

          .option("compression", "gzip")

          .save(temp_path)
Command is running successfully but when I download the file I see for 1 file for each record of the table inside the zipped folder.

Note: Without compression, file is exported as expected.

[bianca_unifeye]

You’re not doing anything “wrong” in the write itself, this is mostly about how Spark writes files vs. how we download them from the UI.

As a workaround write without compression first, then compress.

[iyashk-DB]

It sounds like Spark is splitting your output into many small files (one per row) despite coalesce(1). Can you try setting spark.sql.files.maxRecordsPerFile , this limits how many records can be written into a single output file; if this is set to 1 (or any positive number), Spark will create a new file each time the limit is reached, regardless of partition count from coalesce()

(table.coalesce(1)
      .write
      .mode("overwrite")
      .format(file_format)            # likely "csv"
      .option("header", "true")
      .option("delimiter", field_delimiter)
      .option("compression", "gzip")
      .option("maxRecordsPerFile", 0) # disable row-per-file split
      .save(temp_path))

 But can you be more specific on the issue?