spark data frame parquet vs delta : rows Doesn't match

User16826994223
Databricks Employee
Databricks Employee

‎06-25-2021 10:33 AM

I have data written in Delta on ADLS. As I understand the delta also internal file in parquet format but when Iread the file in different format I got different record count

spark.read.parquet()

 or 

spark.read.format('delta').load()

df = spark.read.format('delta').load("data")
df.count()
> 200000
 
df = spark.read.parquet("data")
df.count()
> 400000

As you can see the difference is quite big.

Is there something I misunderstood about delta vs parquet?

Labels:

User16826994223
Databricks Employee
Databricks Employee

‎06-25-2021 10:34 AM

I think you have written in delta twice using overwrite mode

 .But Delta is versioned data format - when you use 

overwrite

, it doesn't delete previous data, it just writes new files, and don't delete files immediately - they are just marked as deleted in the manifest file that Delta uses. And when you read from Delta, it knows which files are deleted, or not, and read only actual data. Actual deletion of the data files happens when you're performing VACUUM on Delta lake.

But when you read with Parquet, it doesn't have information about deleted files, so it reads everything that you have in directory, so you get twice as many rows.

0 Kudos