Multiple DLT pipelines same target table

[EDDatabricks]

Is it possible to have multiple DLT pipelines write data concurrently and in append mode to the same Delta table? Because of different data sources, with different data volumes and required processing, we would like to have different pipelines streaming from the aforementioned data sources.

We tried setting the "pipelines.tableManagedByMultiplePipelinesCheck.enabled" pipeline configuration to "false" and we managed to stream data to the same target from two different pipelines. However, this configuration option could result in unexpected behavior. Could you please elaborate on this "unexpected bahavior"? What could we expect if multiple pipelines stream data to the same Delta table while having the "pipelines.tableManagedByMultiplePipelinesCheck.enabled" pipeline configuration set to "false"?

[Kaniz_Fatma]

Hi @EDDatabricks,

 Multiple DLT pipelines can write data concurrently and in append mode to the same Delta table.
- Setting "pipelines.tableManagedByMultiplePipelinesCheck.enabled" to "false" allows multiple pipelines to write to the same table.
- However, this can lead to conflicts and data integrity issues due to Delta Lake's Optimistic Concurrency Control (OCC).
- Delta Lake uses OCC to provide ACID transaction guarantees.
- Concurrent transactions read the latest version of the table and make changes optimistically using copy-on-write operations.
- Transactions attempt to atomically commit changes in the Delta log after checking for conflicts.
- Logical conflicts occur when there is a data dependency between two transactions.
- Physical conflicts occur when two transactions modify different rows in the same file.
- Conflicts between transactions can lead to data integrity issues, such as inconsistencies or failed transactions.