saveAsTable sometimes works sometimes dont

[RobDineen]

I have the following Spark (Save As Table) example.  sometimes it works fine, sometimes it fails

Code below with file listed in "/temp" directory. 

This has worked fine as it is, but when I have to create a new Cluster, as I am using the community edition.

It fails, is there a transaction log I have to Vacuum or any other areas I need to clean up first?

Any help would be appreciated

Thank you all

[-werners-]

if you use saveAsTable without any .format() option, it will be saved as a delta lake table.
You also do not specify a location so it gets saved to the default location.  I don't know what that is on community edition, but if something is already present with the same name, the write will fail.
You can try to use .mode("overwrite") with the write.
Or clean up the location first (that is of course only an option for test purposes).

[RobDineen]

I have updated the location so it is explicit. 

but if I use mode("overwrite") that fails as well. 

[-werners-]

the overwrite goes after the write.
What you can do is this:
First write to a new table like testTable1 (delta lake).
The run the write again using write.mode("overwrite")... on the same table, should work.