saveAsTable sometimes works sometimes dont
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 07:25 AM - edited 11-07-2024 07:27 AM
I have the following Spark (Save As Table) example. sometimes it works fine, sometimes it fails
Code below with file listed in "/temp" directory.
This has worked fine as it is, but when I have to create a new Cluster, as I am using the community edition.
It fails, is there a transaction log I have to Vacuum or any other areas I need to clean up first?
Any help would be appreciated
Thank you all
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 07:29 AM
if you use saveAsTable without any .format() option, it will be saved as a delta lake table.
You also do not specify a location so it gets saved to the default location. I don't know what that is on community edition, but if something is already present with the same name, the write will fail.
You can try to use .mode("overwrite") with the write.
Or clean up the location first (that is of course only an option for test purposes).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 07:42 AM
I have updated the location so it is explicit.
but if I use mode("overwrite") that fails as well.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2024 07:51 AM
the overwrite goes after the write.
What you can do is this:
First write to a new table like testTable1 (delta lake).
The run the write again using write.mode("overwrite")... on the same table, should work.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-11-2024 03:15 AM
This seems to work, along with explicitly dropping the Database, and re running all code within Notebook.
Thank you