09-23-2021 03:06 PM
Hi All,
I have a daily spark job that reads and joins 3-4 source tables and writes the df in a parquet format. This data frame consists of 100+ columns. As this job run daily, our deduplication logic identifies the latest record from each of source tables , joins them and eventually overwrites the existing parquet file.
The question becomes - is there a way to implement the incremental write only in cases of a new record or changes in the values in the existing record of the file.
09-27-2021 04:09 AM
the MERGE functionality of delta lake is what you are looking for.
https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html
09-23-2021 10:38 PM
Hi @ Nazar! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the community have an answer to your question first. Or else I will follow up with my team and get back to you soon.Thanks.
09-24-2021 11:19 AM
Thanks, Appreciate the quick response.
09-24-2021 09:12 PM
You're most welcome @Nazar Shaik .
09-27-2021 04:09 AM
the MERGE functionality of delta lake is what you are looking for.
https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html
09-27-2021 02:55 PM
Thanks werners
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.