topic saveAsTable sometimes works sometimes dont in Databricks Free Edition Help

saveAsTable sometimes works sometimes dont

RobDineen — Thu, 07 Nov 2024 15:27:49 GMT

I have the following Spark (Save As Table) example. sometimes it works fine, sometimes it fails

Code below with file listed in "/temp" directory.

This has worked fine as it is, but when I have to create a new Cluster, as I am using the community edition.

It fails, is there a transaction log I have to Vacuum or any other areas I need to clean up first?

Any help would be appreciated

Thank you all

Re: saveAsTable sometimes works sometimes dont

-werners- — Thu, 07 Nov 2024 15:29:57 GMT

if you use saveAsTable without any .format() option, it will be saved as a delta lake table.
You also do not specify a location so it gets saved to the default location. I don't know what that is on community edition, but if something is already present with the same name, the write will fail.
You can try to use .mode("overwrite") with the write.
Or clean up the location first (that is of course only an option for test purposes).

Re: saveAsTable sometimes works sometimes dont

RobDineen — Thu, 07 Nov 2024 15:42:07 GMT

I have updated the location so it is explicit.

but if I use mode("overwrite") that fails as well.

Re: saveAsTable sometimes works sometimes dont

-werners- — Thu, 07 Nov 2024 15:51:05 GMT

the overwrite goes after the write.
What you can do is this:
First write to a new table like testTable1 (delta lake).
The run the write again using write.mode("overwrite")... on the same table, should work.

Re: saveAsTable sometimes works sometimes dont

RobDineen — Mon, 11 Nov 2024 11:15:45 GMT

This seems to work, along with explicitly dropping the Database, and re running all code within Notebook.

Thank you