Py4JJavaError when trying to write dataframe to delta table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-21-2022 09:45 PM
I'm trying to write a dataframe to a delta table and am getting this error.
Im not sure what the issue is as I had no problem successfully writing other dataframes to delta tables. I attached a snippet of the data as well along with the schema:
- Labels:
-
Dataframe
-
Py4jjavaerror
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-21-2022 10:58 PM
hey @Ashad Khan refer this link https://bigdataprogrammers.com/write-dataframe-to-delta-table-in-databricks-with-overwrite-mode/
that might help you ,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-22-2022 01:20 AM
Does your delta tables contains all columns what your dataframe contains. If there is schema mismatch it might be a reason for failure.
df.write.format("delta") \
.option("mergeSchema", "true") \
.mode("append") \
.save(DELTALAKE_SILVER_PATH)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-22-2022 10:35 AM
My understanding was that I was writing to a brand new delta table that doesn't exist, and hence there would be no schema mismatch. I also tried the inferSchema option. Is that not the case? I did the same for other dataframes and it wasn't an issue

