What happens when you write to same Delta table from two different clusters at the same time?

Anonymous — Fri, 18 Jun 2021 21:14:30 GMT

Re: What happens when you write to same Delta table from two different clusters at the same time?

Ryan_Chynoweth — Fri, 18 Jun 2021 21:40:01 GMT

Delta Lake uses optimistic concurrency control to provide transactional guarantees between writes. Under this mechanism, writes operate in three stages:

Read: Reads (if needed) the latest available version of the table to identify which files need to be modified (that is, rewritten).
Write: Stages all the changes by writing new data files.
Validate and commit: Before committing the changes, checks whether the proposed changes conflict with any other changes that may have been concurrently committed since the snapshot that was read. If there are no conflicts, all the staged changes are committed as a new versioned snapshot, and the write operation succeeds. However, if there are conflicts, the write operation fails with a concurrent modification exception rather than corrupting the table as would happen with the write operation on a Parquet table.

Check out this documentation.

topic Re: What happens when you write to same Delta table from two different clusters at the same time? in Data Engineering

What happens when you write to same Delta table from two different clusters at the same time?

Re: What happens when you write to same Delta table from two different clusters at the same time?