I got the same error when I save my dataframe to S3,

Although other DataFrame can save successfully.

I found a method to avoid the problem in my case.

.

Define a function to save the DataFrame to hdfs first,

and then use the saved parquet file create a new DataFrame.

After this the new DataFrame can save to S3 successfully.

.

def save_to_hdfs_first(id, df_save):

　　df_save.write.mode('overwrite').parquet('/tmp/' + id + '.parquet')

　　df_new = spark.read.parquet('/tmp/' + id + '.parquet')

　　return df_new

.

I don't know if the memory or the partition problem,

But this method can indeed solved my problem.