spark data frame parquet vs delta : rows Doesn't match

User16826994223 — Fri, 25 Jun 2021 17:33:41 GMT

I have data written in Delta on ADLS. As I understand the delta also internal file in parquet format but when Iread the file in different format I got different record count

spark.read.parquet()

spark.read.format('delta').load()

df = spark.read.format('delta').load("data")
df.count()
> 200000
 
df = spark.read.parquet("data")
df.count()
> 400000

As you can see the difference is quite big.

Is there something I misunderstood about delta vs parquet?

Re: spark data frame parquet vs delta : rows Doesn't match

User16826994223 — Fri, 25 Jun 2021 17:34:02 GMT

I think you have written in delta twice using overwrite mode

.But Delta is versioned data format - when you use

overwrite

, it doesn't delete previous data, it just writes new files, and don't delete files immediately - they are just marked as deleted in the manifest file that Delta uses. And when you read from Delta, it knows which files are deleted, or not, and read only actual data. Actual deletion of the data files happens when you're performing VACUUM on Delta lake.

But when you read with Parquet, it doesn't have information about deleted files, so it reads everything that you have in directory, so you get twice as many rows.

topic spark data frame parquet vs delta : rows Doesn't match in Data Engineering

spark data frame parquet vs delta : rows Doesn't match

Re: spark data frame parquet vs delta : rows Doesn't match