topic Re: concurrent update to same hive or deltalake table in Data Engineering

concurrent update to same hive or deltalake table

Autel — Sun, 09 Jan 2022 05:31:05 GMT

HI,

I'm interested to know if multiple executors to append the same hive table using saveAsTable or insertInto sparksql. will that cause any data corruption? What configuration do I need to enable concurrent write to same hive table?

what about the same question for deltalake ?

Re: concurrent update to same hive or deltalake table

-werners- — Mon, 10 Jan 2022 09:21:46 GMT

The Hive table will not like this, as the underlying data is parquet format which is not ACID compliant.

Delta lake however is:

https://docs.delta.io/0.5.0/concurrency-control.html

You can see that inserts do not give conflicts.

Re: concurrent update to same hive or deltalake table

Autel — Fri, 14 Jan 2022 01:00:47 GMT

Thanks for your answer.

I found the deltalake on s3 has the following warning on the aws page.

"Warning

Concurrent writes to the same Delta table from multiple Spark drivers can lead to data loss."

For single driver with multiple executors, will concurrent write to the same table be an issue as well?

Re: concurrent update to same hive or deltalake table

-werners- — Fri, 14 Jan 2022 08:03:26 GMT

No because that is how spark works.

The driver defines which worker writes what and is up to speed with what is going on.

That is also the reason that multiple drivers (read multiple spark programs) can give conflicts as the drivers do not know of each other what they are doing.