topic Re: Multiple DLT Pipelines Sharing a Single Cluster in Data Engineering

Multiple DLT Pipelines Sharing a Single Cluster

bozhu — Fri, 07 Oct 2022 21:41:19 GMT

When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.

I understand DLT pipelines utilising their own special clusters, so it will still be a good solution that a DLT type cluster can be defined in a Workflows Job and this cluster is shared across different DLT pipeline tasks.

The biggest benefit of achieving this will be a significant saving of wasted individual DLT pipeline cluster start time.

Re: Multiple DLT Pipelines Sharing a Single Cluster

Hubert-Dudek — Sun, 16 Oct 2022 10:28:38 GMT

This is impossible, but in DLT pipeline settings, you can add multiple files to be processed in one pipeline.

I hope that the serverless version will bring potential savings in the future.

Re: Multiple DLT Pipelines Sharing a Single Cluster

Anonymous — Thu, 17 Nov 2022 05:17:12 GMT

Hi @Bo Zhu

Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!