Limited concurrent running DLT's within a pipeline

JulianKrüger
New Contributor

‎01-23-2025 02:07 AM

Hi Champions!

We are currently experiencing a few unexplainable limitations when executing pipelines with > 50 DLT tables. It looks like, that there is some calculation in the background in place, to determine the maximum number of concurrent running DLT's - in our case 16

Pipeline Config:
Cloud: Azure Databricks
Product Edition: Pro
Channel: Current
Pipeline mode: Triggered
Storage option: Unity Catalog

Initial Cluster Config:
Cluster Policy: None
Cluster mode: Enhanced autoscaling
Min / Max Worker: 1/6
Photon enabled
Worker / Driver type: Standard_F8

Analyze the Issue:
I checked the detail column for the event_type "cluster_resources" in the event_log tvf for the pipeline. The value for "num_task_slots" is limited to 16. This was a first indicator for me, to think, that this number influence the maximum concurrent running DLT's.

JulianKrger_0-1737625978665.png

Tries to increase the Concurrency:

  • Changed the Worker and Driver Types
  • Increased the Driver and Worker Size
  • Changed the product edition
  • Enabled and disabled Photon
  • All Cluster modes (Fixed size, legacy autoscaling, enhanced autoscaling)

One configuration worked to increase the "num_task_slots". Set the minimum number of workers to 5:

JulianKrger_1-1737626627178.png

After that change, the "num_task_slots" increased to 40.
(I cannot realy derive, why the num_task_slots for 4 worker is 16 and for 5 worker it is 40)

Concurrency still limited:
When I extend my query against the event_log and calculate the number of concurrent running pipelines, the pipeline still execute a maximum of 16 pipelines at the same time:

JulianKrger_2-1737626698802.png

So maybe the "num_task_slots" to not influence the number of concurrent running DLT's?

Open questions:

  • Does the “num_task_slots” have anything to do with the maximum parity?
    • If yes:
      • Why the pipeline still caps at 16 concurrent running DLT's while the number has increased to 40?
      • How does Databricks calculate the "num_task_slots"?
    • If no:
      • What does the number "num_task_slots" tell me?
      • What else determine the maximum of concurrent running DLT's?
  • How can I increase the number of concurrent running pipelines? Is there any cluster conf or DLT conf I can provide?

I do not found any limitation or calculation out there to answer that questions.

I hope that one of you champions out there can help me with this.


Best regards

Julian

0 Kudos