Unable to configure custom compute for DLT pipeline

[GANAPATI_HEGDE]

I am trying to configure cluster for a pipeline like above, However dlt keeps using the small cluster as usual, how to resolve this?

[szymon_dybczak]

Hi @GANAPATI_HEGDE ,

Your YAML definition looks correct. I've used following definition and it worked as expected. Which version of databricks cli are you using? In my case with CLI v0.272.0 I didn't encounter any issues.

[GANAPATI_HEGDE]

i updated my CLI and deployed the job, still i dont see the clusters updates in  pipeline

[stbjelcevic]

Hi @GANAPATI_HEGDE ,

Are you still having issues with this? If so, does your bundle have a targets section? Target overrides define the runtime cluster settings that the pipeline actually uses when deployed to that target and take precedence.

Is there possibly a compute policy constraining the DLT cluster settings? If a compute policy for DLT clusters is applied (cluster_type = dlt), its constraints (like worker type and min/max workers) take precedence over your YAML cluster settings.