szymon_dybczak
Esteemed Contributor III

Definitely weird, basically since you're writing in delta format you shouldn't lose any data. Delta tables support ACID transaction, so either it writes to your table all data or it leaves table in consistent state (before write action).
In your situation, I would run df.count() before wrtiting to delta table to double check that dataframe contains expected number of rows.
You can also get rid of persist from above code, it's useful if you're going to use the same dataframe in different set of calculation. No use for that just before writing to the table.