topic Re: DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute in Get Started Discussions

DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute

ChristianRRL — Tue, 04 Jun 2024 20:28:42 GMT

Hi there, I would like to understand for DLT jobs if there is any way to get DLT jobs running in an *existing* (and currently running) All-purpose compute rather than spinning up an "ephemeral" (not yet initialized) Job Compute?

Re: DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute

raphaelblg — Tue, 04 Jun 2024 20:57:15 GMT

Hi @ChristianRRL , you cannot run a DLT pipeline in an All-purpose compute cluster.

DLT Clusters are managed resources. Here are some details on how to configure a DLT cluster: https://docs.databricks.com/en/delta-live-tables/settings.html#configure-your-compute-settings.

Re: DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute

ChristianRRL — Tue, 04 Jun 2024 21:00:19 GMT

I suspected this might be the case. Thank you for your confirmation!

Re: DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute

ChristianRRL — Tue, 04 Jun 2024 21:11:14 GMT

@raphaelblg Actually, one follow-up question. Very brief out-loud thought, why doesn't Databricks at least offer the option to use our existing All-purpose compute clusters? If my understanding is correct, DLT Clusters are meant to run jobs more efficiently (and therefore cheaper) than the traditional all-purpose clusters, but if we already have dedicated all-purpose clusters running without the possibility of letting our DLT jobs run on those all-purpose clusters, then no matter how efficient/cheap DLT runs it will always be an additional cost to our existing setup.

In short, we already have a "baked in" cost of running our all-purpose clusters, but adding DLT will only ever *drive up* costs rather than cut them since there is no way to have them run in our all-purpose clusters.