topic Can multiple streams write to a Delta table at the same time? in Data Engineering

Can multiple streams write to a Delta table at the same time?

User16826992666 — Thu, 17 Jun 2021 03:27:12 GMT

Wondering if there any dangers to doing this, and if it's a best practice. I'm concerned there could be conflicts but I'm not sure how Delta would handle it.

Re: Can multiple streams write to a Delta table at the same time?

Ryan_Chynoweth — Fri, 18 Jun 2021 00:05:29 GMT

Delta would enforce this dual write using optimistic concurrency and would not have an issue generally accepting multiple writes. Please note that data quality could be an issue if there is duplicate data coming in from multiple streams of if there were conflicting transactions (two updates, deletes etc.).

Re: Can multiple streams write to a Delta table at the same time?

sajith_appukutt — Fri, 18 Jun 2021 05:51:32 GMT

>Can multiple streams write to a Delta table at the same time?

Yes delta uses optimistic concurrency control and configurable isolation levels

>I'm concerned there could be conflicts but I'm not sure how Delta would handle it.

Write operations can result in conflicts depending on the isolation level and could raise an exception. For more details on the types of exceptions and how to handle them - see https://docs.databricks.com/delta/concurrency-control.html#conflict-exceptions