โ01-08-2022 09:31 PM
HI,
I'm interested to know if multiple executors to append the same hive table using saveAsTable or insertInto sparksql. will that cause any data corruption? What configuration do I need to enable concurrent write to same hive table?
what about the same question for deltalake ?
โ01-10-2022 01:21 AM
The Hive table will not like this, as the underlying data is parquet format which is not ACID compliant.
Delta lake however is:
https://docs.delta.io/0.5.0/concurrency-control.html
You can see that inserts do not give conflicts.
โ01-10-2022 01:21 AM
The Hive table will not like this, as the underlying data is parquet format which is not ACID compliant.
Delta lake however is:
https://docs.delta.io/0.5.0/concurrency-control.html
You can see that inserts do not give conflicts.
โ01-13-2022 05:00 PM
Hi
Thanks for your answer.
I found the deltalake on s3 has the following warning on the aws page.
"Warning
Concurrent writes to the same Delta table from multiple Spark drivers can lead to data loss."
For single driver with multiple executors, will concurrent write to the same table be an issue as well?
โ01-14-2022 12:03 AM
No because that is how spark works.
The driver defines which worker writes what and is up to speed with what is going on.
That is also the reason that multiple drivers (read multiple spark programs) can give conflicts as the drivers do not know of each other what they are doing.
โ01-10-2022 09:04 PM
Hi @Weide Zhangโ , Does @[werners] (Customer)โ 's reply answer your question?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group