Update a part of parquet partition is deleting existing data.

JGAICT
New Contributor

‎07-02-2023 11:56 PM

I have 4 months of data and I partitioned it on Year and Month column, so my parquet partition looks like 

JGAICT_1-1688364696457.jpeg

JGAICT_0-1688364651334.png

JGAICT_2-1688364739675.png

Data is present inside each monthly partition folder in parquet format.

Then I loaded data for July and also modified some values in August. After generating the required data, I tried to save the output with same partition (Year,Month) as before this time the data did not have September, October and November entries but only for 2 months. The result is as

JGAICT_3-1688365438030.png

JGAICT_4-1688365625614.png

Interestingly when I look unto the month of September, the parquet files are missing, same in October and November. The folders have some modifications based on last modified.

JGAICT_6-1688365838569.png

JGAICT_5-1688365699622.png

I am guessing that the files had gone missing because the 2nd write had overridden the 1st one because the partition was also on year. Is there a way to overcome this problem and avoid files to be deleted?

0 Kudos