DLT pipelines in the same job sharing compute

[bradleyjamrozik]

If I have a job like this that orchestrates N DLT pipelines, what setting do I need to adjust so that they use the same compute resources between steps rather than spinning up and shutting down for each individual pipeline?

[bradleyjamrozik]

Edit: I'm using databricks asset bundles to deploy both the pipelines and jobs.

[shan_chandra]

@bradleyjamrozik  - under the DLT settings, notebooks can be listed all together. It will deploy a single compute resource for all the tasks. 

[bradleyjamrozik]

The different pipelines point to different catalogs/schemas so they have to be separated out.

[shan_chandra]

@bradleyjamrozik - I checked internally too. DLT pipelines cannot share resources. 

[RasmusBrostroem]

@shan_chandra Hello, I have the same issue, where I have a job that uses serverless compute and in this job I do some different tasks and then I start a DLT pipeline, which also uses serverless compute and this means that the job again have to wait for a resource to start. 

Has anything changed since 2023 and if not, then is it something that will be possible in the future? For my use case it really limits the possibilities, since for the overall job there are now 10 minutes of where resources have to start, which isn't optimal, when this job has to run each hour and has to provide results quickly.