Multiple DLT Pipelines Sharing a Single Cluster
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-07-2022 02:41 PM
When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.
I understand DLT pipelines utilising their own special clusters, so it will still be a good solution that a DLT type cluster can be defined in a Workflows Job and this cluster is shared across different DLT pipeline tasks.
The biggest benefit of achieving this will be a significant saving of wasted individual DLT pipeline cluster start time.
Labels:
- Labels:
-
DLT
-
DLT Pipelines