cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Py4JJavaError when trying to write dataframe to delta table

AK98
New Contributor II

I'm trying to write a dataframe to a delta table and am getting this error.

image.pngIm not sure what the issue is as I had no problem successfully writing other dataframes to delta tables. I attached a snippet of the data as well along with the schema:

imageimage

4 REPLIES 4

Rishabh264
Honored Contributor II

ramravi
Contributor II

Does your delta tables contains all columns what your dataframe contains. If there is schema mismatch it might be a reason for failure.

df.write.format("delta") \
           .option("mergeSchema", "true") \
           .mode("append") \
           .save(DELTALAKE_SILVER_PATH)

AK98
New Contributor II

My understanding was that I was writing to a brand new delta table that doesn't exist, and hence there would be no schema mismatch. I also tried the inferSchema option. Is that not the case? I did the same for other dataframes and it wasn't an issue

Kaniz
Community Manager
Community Manager

Hi @Ravi Teja​, We haven’t heard from you since the last response from @Rishabh Pandey​​​, and I was checking back to see if his suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.