cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute

ChristianRRL
Contributor III

Hi there, I would like to understand for DLT jobs if there is any way to get DLT jobs running in an *existing* (and currently running) All-purpose compute rather than spinning up an "ephemeral" (not yet initialized) Job Compute?  

ChristianRRL_0-1717532769488.png

 

3 REPLIES 3

raphaelblg
Honored Contributor II

Hi @ChristianRRL , you cannot run a DLT pipeline in an All-purpose compute cluster.

DLT Clusters are managed resources. Here are some details on how to configure a DLT cluster: https://docs.databricks.com/en/delta-live-tables/settings.html#configure-your-compute-settings.

 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

I suspected this might be the case. Thank you for your confirmation!

@raphaelblg  Actually, one follow-up question. Very brief out-loud thought, why doesn't Databricks at least offer the option to use our existing All-purpose compute clusters? If my understanding is correct, DLT Clusters are meant to run jobs more efficiently (and therefore cheaper) than the traditional all-purpose clusters, but if we already have dedicated all-purpose clusters running without the possibility of letting our DLT jobs run on those all-purpose clusters, then no matter how efficient/cheap DLT runs it will always be an additional cost to our existing setup.

In short, we already have a "baked in" cost of running our all-purpose clusters, but adding DLT will only ever *drive up* costs rather than cut them since there is no way to have them run in our all-purpose clusters.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group