Hi @nafri A​ 

I second what @Jose Gonzalez​ suggested. You can first read in spark and convert it into Delta lake, after conversion, you can run OPTIMIZE command which will coalesce all the small files into a bigger sizes. Then you can just convert it back to parquet from Delta if you want. Below are the commands that will be useful:

Convert Parquet to Delta lake:

%sql
-- Convert non partitioned parquet table at path '<path-to-table>'
 CONVERT TO DELTA parquet.`<path-to-table>`
 
-- Convert partitioned Parquet table at path '<path-to-table>' and partitioned by integer columns named 'part' and 'part2'
CONVERT TO DELTA parquet.`<path-to-table>` PARTITIONED BY (part int, part2 int)

Optimize Delta Lake:

%sql
-- Optimize our Delta table 
OPTIMIZE delta.`/table_delta/`

Convert back to Parquet:

1. If you have performed Delta Lake operations that can change the data files (for example, delete or merge), run VACUUM with a retention of 0 hours to delete all data files that do not belong to the latest version of the table.
2. Delete the _delta_log directory in the table directory. 

Note: This will be a one time process to compact small files into bigger files