topic DLT concurrent pipeline updates. in Data Engineering

DLT concurrent pipeline updates.

ElaPG — Fri, 01 Dec 2023 11:28:00 GMT

Hi!

Regarding this info "An Azure Databricks workspace is limited to 100 concurrent pipeline updates." (Release 2023.16 - Azure Databricks | Microsoft Learn), what is considered as an update? Changes in pipeline logic or each pipeline run?

Re: DLT concurrent pipeline updates.

ElaPG — Mon, 04 Dec 2023 08:15:15 GMT

First of all, thank you for your reply.

So if I get 99 new files that should trigger DLT pipeline update and at the same time I change something in pipeline logic it means that all these activities contribute to 100 concurrent updates (if I understood correctly).