spark data frame parquet vs delta : rows Doesn't match
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 10:33 AM
I have data written in Delta on ADLS. As I understand the delta also internal file in parquet format but when Iread the file in different format I got different record count
spark.read.parquet()
or
spark.read.format('delta').load()
df = spark.read.format('delta').load("data")
df.count()
> 200000
df = spark.read.parquet("data")
df.count()
> 400000As you can see the difference is quite big.
Is there something I misunderstood about delta vs parquet?
Labels:
- Labels:
-
Dataframe
-
Different Number