- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-30-2025 12:06 AM
Hi @JulianKrüger ,
• The "num_task_slots" parameter in Databricks Delta Live Tables (DLT) pipelines is related to the concurrency of tasks within a pipeline. It determines the number of concurrent tasks that can be executed. However, this parameter does not directly determine the maximum number of concurrent running DLT pipelines within a workspace.
• A pipeline might still be capped at 16 concurrent running DLTs even if the "num_task_slots" has been increased to 40 due to other limitations or configurations in the system, such as cluster configurations or workspace-level limits that are not directly influenced by the "num_task_slots" parameter.
• The "num_task_slots" is calculated based on the available resources and the specific configurations of the cluster, such as the number of workers and the instance types used. Enhanced autoscaling and other cluster settings can also impact how resources are allocated for task slots.