Py4JJavaError when trying to write dataframe to delta table

AK98
New Contributor II

‎12-21-2022 09:45 PM

I'm trying to write a dataframe to a delta table and am getting this error.

image.pngIm not sure what the issue is as I had no problem successfully writing other dataframes to delta tables. I attached a snippet of the data as well along with the schema:

imageimage

Labels:
Preview file
15 KB
Preview file
67 KB
Preview file
26 KB
0 Kudos

Rishabh-Pandey
Databricks MVP

‎12-21-2022 10:58 PM

hey @Ashad Khan​ refer this link https://bigdataprogrammers.com/write-dataframe-to-delta-table-in-databricks-with-overwrite-mode/

that might help you ,

Rishabh Pandey
0 Kudos

ramravi
Contributor II

‎12-22-2022 01:20 AM

Does your delta tables contains all columns what your dataframe contains. If there is schema mismatch it might be a reason for failure.

df.write.format("delta") \
           .option("mergeSchema", "true") \
           .mode("append") \
           .save(DELTALAKE_SILVER_PATH)

0 Kudos

AK98
New Contributor II
In response to ramravi

‎12-22-2022 10:35 AM

My understanding was that I was writing to a brand new delta table that doesn't exist, and hence there would be no schema mismatch. I also tried the inferSchema option. Is that not the case? I did the same for other dataframes and it wasn't an issue

0 Kudos