Databricks Community

@Reka, thank you. I have exception in converting data type. Hopefully the issue won't happen again after the fix.

@ReKa​  thank heaps. I'm using S3a already, and schema is very clearly defined and look like this:OUTPUT_SCHEMA = StructType([ StructField("c1", StringType(), True), StructField("c2", ArrayType(StringType()), True), StructField("c3", S...

Hi, I get this exception when using df.write.parquet(), on both overwrite and default mode, for completely new location. The exception is intermittent and causes our data pipeline crash randomly. Spark version: 1.6.0 Has anyone have more information...