cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

concurrent update to same hive or deltalake table

Autel
New Contributor II

HI,

I'm interested to know if multiple executors to append the same hive table using saveAsTable or insertInto sparksql. will that cause any data corruption? What configuration do I need to enable concurrent write to same hive table?

what about the same question for deltalake ?

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

The Hive table will not like this, as the underlying data is parquet format which is not ACID compliant.

Delta lake however is:

https://docs.delta.io/0.5.0/concurrency-control.html

You can see that inserts do not give conflicts.

View solution in original post

4 REPLIES 4

-werners-
Esteemed Contributor III

The Hive table will not like this, as the underlying data is parquet format which is not ACID compliant.

Delta lake however is:

https://docs.delta.io/0.5.0/concurrency-control.html

You can see that inserts do not give conflicts.

Autel
New Contributor II

Hi

Thanks for your answer.

I found the deltalake on s3 has the following warning on the aws page.

"Warning

Concurrent writes to the same Delta table from multiple Spark drivers can lead to data loss."

For single driver with multiple executors, will concurrent write to the same table be an issue as well?

-werners-
Esteemed Contributor III

No because that is how spark works.

The driver defines which worker writes what and is up to speed with what is going on.

That is also the reason that multiple drivers (read multiple spark programs) can give conflicts as the drivers do not know of each other what they are doing.

Kaniz
Community Manager
Community Manager

Hi @Weide Zhang​ , Does @[werners] (Customer)​ 's reply answer your question?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!