Multiple DLT pipelines same target table

EDDatabricks — Mon, 25 Sep 2023 13:54:32 GMT

Is it possible to have multiple DLT pipelines write data concurrently and in append mode to the same Delta table? Because of different data sources, with different data volumes and required processing, we would like to have different pipelines streaming from the aforementioned data sources.

We tried setting the "pipelines.tableManagedByMultiplePipelinesCheck.enabled" pipeline configuration to "false" and we managed to stream data to the same target from two different pipelines. However, this configuration option could result in unexpected behavior. Could you please elaborate on this "unexpected bahavior"? What could we expect if multiple pipelines stream data to the same Delta table while having the "pipelines.tableManagedByMultiplePipelinesCheck.enabled" pipeline configuration set to "false"?

Re: Multiple DLT pipelines same target table

claudiayuan — Sun, 24 Nov 2024 10:10:56 GMT

hello! did you get the answer?

Re: Multiple DLT pipelines same target table

arvind_c — Tue, 25 Feb 2025 13:29:05 GMT

Hi I wanted to understand how you set "pipelines.tableManagedByMultiplePipelinesCheck.enabled" to true? I'm not able to figure out how and where you set this param

topic Re: Multiple DLT pipelines same target table in Data Engineering

Multiple DLT pipelines same target table

Re: Multiple DLT pipelines same target table

Re: Multiple DLT pipelines same target table