@ReKa​  thank heaps.

I'm using S3a already, and schema is very clearly defined and look like this:

OUTPUT_SCHEMA = StructType([ 
    StructField("c1", StringType(), True), 
    StructField("c2", ArrayType(StringType()), True), 
    StructField("c3", ShortType(), True), 
    StructField("c4", BooleanType(), True) 
])

I think this schema is tight enough.

On the notes:

+ "Compute Optimised had problems": do you what types of problems it has? or only writing data?

+ json and converting to parquet at the end (But watch for zero-sized files which can show corrupt partitions): do you have more information about this?

Many thanks.