Multiple DLT Pipelines Sharing a Single Cluster

[bozhu]

When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.

I understand DLT pipelines utilising their own special clusters, so it will still be a good solution that a DLT type cluster can be defined in a Workflows Job and this cluster is shared across different DLT pipeline tasks.

The biggest benefit of achieving this will be a significant saving of wasted individual DLT pipeline cluster start time.

[Hubert-Dudek]

This is impossible, but in DLT pipeline settings, you can add multiple files to be processed in one pipeline.

I hope that the serverless version will bring potential savings in the future.

[Anonymous]

Hi @Bo Zhu​ 

Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!