Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-25-2024 10:18 AM
Hello @Riccardo96,
This behavior suggests that rows might be getting dropped or overwritten during the writing process, particularly when using the replaceWhere option with clustering or partitioning.
The replaceWhere option replaces data based on the specified condition (year, month, and day). If multiple files have overlapping data for the same day, some rows might get overwritten
You can debug this by running the before and after writing:
df_adobe_nav_utente.groupBy("year", "month", "day").count().show()