topic Py4JJavaError when trying to write dataframe to delta table in Data Engineering

Py4JJavaError when trying to write dataframe to delta table

AK98 — Thu, 22 Dec 2022 05:45:46 GMT

I'm trying to write a dataframe to a delta table and am getting this error.

Im not sure what the issue is as I had no problem successfully writing other dataframes to delta tables. I attached a snippet of the data as well along with the schema:

Re: Py4JJavaError when trying to write dataframe to delta table

Rishabh-Pandey — Thu, 22 Dec 2022 06:58:27 GMT

hey @Ashad Khan refer this link https://bigdataprogrammers.com/write-dataframe-to-delta-table-in-databricks-with-overwrite-mode/

that might help you ,

Re: Py4JJavaError when trying to write dataframe to delta table

ramravi — Thu, 22 Dec 2022 09:20:54 GMT

Does your delta tables contains all columns what your dataframe contains. If there is schema mismatch it might be a reason for failure.

df.write.format("delta") \
           .option("mergeSchema", "true") \
           .mode("append") \
           .save(DELTALAKE_SILVER_PATH)

Re: Py4JJavaError when trying to write dataframe to delta table

AK98 — Thu, 22 Dec 2022 18:35:18 GMT

My understanding was that I was writing to a brand new delta table that doesn't exist, and hence there would be no schema mismatch. I also tried the inferSchema option. Is that not the case? I did the same for other dataframes and it wasn't an issue