Multiple DLT Pipelines Sharing a Single Cluster

bozhu
Contributor

When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.

I understand DLT pipelines utilising their own special clusters, so it will still be a good solution that a DLT type cluster can be defined in a Workflows Job and this cluster is shared across different DLT pipeline tasks.

The biggest benefit of achieving this will be a significant saving of wasted individual DLT pipeline cluster start time.