cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute

ChristianRRL
Contributor II

Hi there, I would like to understand for DLT jobs if there is any way to get DLT jobs running in an *existing* (and currently running) All-purpose compute rather than spinning up an "ephemeral" (not yet initialized) Job Compute?  

ChristianRRL_0-1717532769488.png

 

1 ACCEPTED SOLUTION

Accepted Solutions

raphaelblg
Honored Contributor
Honored Contributor

Hi @ChristianRRL , you cannot run a DLT pipeline in an All-purpose compute cluster.

DLT Clusters are managed resources. Here are some details on how to configure a DLT cluster: https://docs.databricks.com/en/delta-live-tables/settings.html#configure-your-compute-settings.

 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

View solution in original post

3 REPLIES 3

raphaelblg
Honored Contributor
Honored Contributor

Hi @ChristianRRL , you cannot run a DLT pipeline in an All-purpose compute cluster.

DLT Clusters are managed resources. Here are some details on how to configure a DLT cluster: https://docs.databricks.com/en/delta-live-tables/settings.html#configure-your-compute-settings.

 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

I suspected this might be the case. Thank you for your confirmation!

@raphaelblg  Actually, one follow-up question. Very brief out-loud thought, why doesn't Databricks at least offer the option to use our existing All-purpose compute clusters? If my understanding is correct, DLT Clusters are meant to run jobs more efficiently (and therefore cheaper) than the traditional all-purpose clusters, but if we already have dedicated all-purpose clusters running without the possibility of letting our DLT jobs run on those all-purpose clusters, then no matter how efficient/cheap DLT runs it will always be an additional cost to our existing setup.

In short, we already have a "baked in" cost of running our all-purpose clusters, but adding DLT will only ever *drive up* costs rather than cut them since there is no way to have them run in our all-purpose clusters.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!