concurrent update to same hive or deltalake table

[Autel]

HI,

I'm interested to know if multiple executors to append the same hive table using saveAsTable or insertInto sparksql. will that cause any data corruption? What configuration do I need to enable concurrent write to same hive table?

what about the same question for deltalake ?

[-werners- · Accepted solution]

The Hive table will not like this, as the underlying data is parquet format which is not ACID compliant.

Delta lake however is:

https://docs.delta.io/0.5.0/concurrency-control.html

You can see that inserts do not give conflicts.

[Autel]

Hi

Thanks for your answer.

I found the deltalake on s3 has the following warning on the aws page.

"Warning

Concurrent writes to the same Delta table from multiple Spark drivers can lead to data loss."

For single driver with multiple executors, will concurrent write to the same table be an issue as well?

[-werners-]

No because that is how spark works.

The driver defines which worker writes what and is up to speed with what is going on.

That is also the reason that multiple drivers (read multiple spark programs) can give conflicts as the drivers do not know of each other what they are doing.