Unable to configure custom compute for DLT pipeline

GANAPATI_HEGDE
New Contributor III

‎11-09-2025 10:01 PM

GANAPATI_HEGDE_0-1762754316899.png

I am trying to configure cluster for a pipeline like above, However dlt keeps using the small cluster as usual, how to resolve this?

GANAPATI_HEGDE_1-1762754398253.png

 

Labels:
0 Kudos

szymon_dybczak
Esteemed Contributor III

‎11-10-2025 04:38 AM

Hi @GANAPATI_HEGDE ,

Your YAML definition looks correct. I've used following definition and it worked as expected. Which version of databricks cli are you using? In my case with CLI v0.272.0 I didn't encounter any issues.

szymon_dybczak_0-1762778206958.png

 

0 Kudos

GANAPATI_HEGDE
New Contributor III

‎11-10-2025 08:41 PM

i updated my CLI and deployed the job, still i dont see the clusters updates in  pipeline

0 Kudos

stbjelcevic
Databricks Employee
Databricks Employee
In response to GANAPATI_HEGDE

‎12-08-2025 01:13 PM

Hi @GANAPATI_HEGDE ,

Are you still having issues with this? If so, does your bundle have a targets section? Target overrides define the runtime cluster settings that the pipeline actually uses when deployed to that target and take precedence.

Is there possibly a compute policy constraining the DLT cluster settings? If a compute policy for DLT clusters is applied (cluster_type = dlt), its constraints (like worker type and min/max workers) take precedence over your YAML cluster settings.

0 Kudos