topic Re: Databricks overwrite didn't delete previous data in Data Engineering

Databricks overwrite didn't delete previous data

阳光彩虹小白马 — Wed, 16 Oct 2024 08:39:34 GMT

Hi databricks, we met an issue like below picture shows:

we use pyspark api to store data into ADLS :

df.write.partitionBy("xx").option("partitionOverwriteMode","dynamic").mode("overwrite").parquet(xx)

However, not sure why the second time we overwrite this partition on 2024-09-26 4:29 PM, the previous data still exists...

The last committed log "_committed_3404689632661433446" shows like below: what has been removed is not tid 3175486376768535369 which was run on 2024-09-26 4:23 PM, what was removed was the data file in tid

2405413862834130470 which was run on 2024-09-20...

Does anyone know the root cause? and how to removed those data which should already be deleted? Thanks!

Re: Databricks overwrite didn't delete previous data

Himanshu6 — Thu, 17 Oct 2024 07:06:54 GMT

Ensure Correct Partition Column Values : Double-check that the values in your partition column "xx" are consistent across the dataset. Make sure there are no formatting issues or null values.

Re: Databricks overwrite didn't delete previous data

Himanshu6 — Thu, 17 Oct 2024 08:19:54 GMT

To more Clarify in Delta lake if you are writing or overwriting some data then it is Creating the new version if you see in table then you will be able to see the new Data but when you check in location the parquet file would be present old and latest.

If your motive is to write data in ADLS and use that parquet file then use .format('parquet')

Re: Databricks overwrite didn't delete previous data

Panda — Thu, 17 Oct 2024 09:24:52 GMT

@阳光彩虹小白马

The issue you're encountering seems to involve inconsistent behavior in partition overwrites using PySpark with ADLS.

Can you validate the below along with what @Himanshu6 mentioned.

Force Spark to refresh the metadata of the data lake directory.
Ensure that the mode(partitionOverwriteMode) is set properly before executing the overwrite operation.