Multiple DLT pipelines same target table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-25-2023 06:54 AM
Is it possible to have multiple DLT pipelines write data concurrently and in append mode to the same Delta table? Because of different data sources, with different data volumes and required processing, we would like to have different pipelines streaming from the aforementioned data sources.
We tried setting the "pipelines.tableManagedByMultiplePipelinesCheck.enabled" pipeline configuration to "false" and we managed to stream data to the same target from two different pipelines. However, this configuration option could result in unexpected behavior. Could you please elaborate on this "unexpected bahavior"? What could we expect if multiple pipelines stream data to the same Delta table while having the "pipelines.tableManagedByMultiplePipelinesCheck.enabled" pipeline configuration set to "false"?
- Labels:
-
Delta Lake
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-24-2024 02:10 AM
hello! did you get the answer?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-25-2025 05:29 AM
Hi I wanted to understand how you set "pipelines.tableManagedByMultiplePipelinesCheck.enabled" to true? I'm not able to figure out how and where you set this param

