I have a fairly small, simple DataFrame, month:

month.schema

org.apache.spark.sql.types.StructType = StructType(StructField(month,DateType,true), StructField(real_month,TimestampType,true), StructField(month_millis,LongType,true))

The month Dataframe is derived from a Dataframe originally created from a RDD which comes from a sc.parallelize(...).

I try to save it as a table:

month.write.mode(SaveMode.Overwrite).saveAsTable("month_x2")

And I get an exception. The root cause seems to be (also see full stacktrace below):

Caused by: java.io.IOException: File already exists: /databricks-prod-storage-virginia/dbc-44061e6b-9dd3/0/user/hive/warehouse/month_x2/part-r-00002-9858e235-1c6c-4276-800d-18c8a760a416.gz.parquet

I've restarted the cluster, and reran the notebook and get the same result every time. I'm using Overwrite mode (although, I think non-overwrite produces a different error anyway). Also, I get this error even when I change the tableName (i.e. even on the first saveAsTable call for a given name).

Error is in attached file