topic Partition By () on Delta Files in Data Engineering

Partition By () on Delta Files

HarshaK — Thu, 07 Apr 2022 08:43:53 GMT

Hi All,

I am trying to Partition By () on Delta file in pyspark language and using command:

df.write.format("delta").mode("overwrite").option("overwriteSchema","true").partitionBy("Partition Column").save("Partition file path") -- It doesnt seems to work for me.

df.write.option("header",True).partitionBy("Partition Column").mode("overwrite").parquet("Partition file path") -- it worked but in the further steps it complains about the file type is not delta.

Please suggest the code to save partition file in delta format.

Thanks in advance.

Re: Partition By () on Delta Files

Hubert-Dudek — Thu, 07 Apr 2022 09:35:47 GMT

.save("Partition file path") - it should be a folder path.

Additionally what runtime are you using? It was like that long before spark 3.x check it with command

sc.version

Re: Partition By () on Delta Files

HarshaK — Thu, 07 Apr 2022 10:52:33 GMT

Thanks @Hubert Dudek it worked.

Re: Partition By () on Delta Files

Hubert-Dudek — Thu, 07 Apr 2022 11:48:52 GMT

Haven't helped much, but I would appreciate if you select my answer as the best one 🙂

Re: Partition By () on Delta Files

Anonymous — Tue, 26 Apr 2022 16:36:14 GMT

Hey @Harsha kriplani

Hope you are well. Thank you for posting in here. It is awesome that you found a solution. Would you like to mark Hubert's answer as best? It would be really helpful for the other members too.

Cheers!