09-23-2021 03:06 PM
Hi All,
I have a daily spark job that reads and joins 3-4 source tables and writes the df in a parquet format. This data frame consists of 100+ columns. As this job run daily, our deduplication logic identifies the latest record from each of source tables , joins them and eventually overwrites the existing parquet file.
The question becomes - is there a way to implement the incremental write only in cases of a new record or changes in the values in the existing record of the file.
09-27-2021 04:09 AM
the MERGE functionality of delta lake is what you are looking for.
https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html
09-23-2021 10:38 PM
Hi @ Nazar! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the community have an answer to your question first. Or else I will follow up with my team and get back to you soon.Thanks.
09-24-2021 11:19 AM
Thanks, Appreciate the quick response.
09-24-2021 09:12 PM
You're most welcome @Nazar Shaik .
09-27-2021 04:09 AM
the MERGE functionality of delta lake is what you are looking for.
https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html
09-27-2021 02:55 PM
Thanks werners
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!