cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Partition By () on Delta Files

HarshaK
New Contributor III

Hi All,

I am trying to Partition By () on Delta file in pyspark language and using command:

df.write.format("delta").mode("overwrite").option("overwriteSchema","true").partitionBy("Partition Column").save("Partition file path") -- It doesnt seems to work for me.

df.write.option("header",True).partitionBy("Partition Column").mode("overwrite").parquet("Partition file path") -- it worked but in the further steps it complains about the file type is not delta.

Please suggest the code to save partition file in delta format.

Thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

.save("Partition file path") - it should be a folder path.

Additionally what runtime are you using? It was like that long before spark 3.x check it with command

sc.version

View solution in original post

4 REPLIES 4

Hubert-Dudek
Esteemed Contributor III

.save("Partition file path") - it should be a folder path.

Additionally what runtime are you using? It was like that long before spark 3.x check it with command

sc.version

HarshaK
New Contributor III

Thanks @Hubert Dudek​  it worked.

Hubert-Dudek
Esteemed Contributor III

Haven't helped much, but I would appreciate if you select my answer as the best one 🙂

Anonymous
Not applicable

Hey @Harsha kriplani​ 

Hope you are well. Thank you for posting in here. It is awesome that you found a solution. Would you like to mark Hubert's answer as best?  It would be really helpful for the other members too.

Cheers!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.