topic Re: Unable to configure custom compute for DLT pipeline in Data Engineering

Unable to configure custom compute for DLT pipeline

GANAPATI_HEGDE — Mon, 10 Nov 2025 06:01:35 GMT

I am trying to configure cluster for a pipeline like above, However dlt keeps using the small cluster as usual, how to resolve this?

Re: Unable to configure custom compute for DLT pipeline

szymon_dybczak — Mon, 10 Nov 2025 12:38:37 GMT

Hi @GANAPATI_HEGDE ,

Your YAML definition looks correct. I've used following definition and it worked as expected. Which version of databricks cli are you using? In my case with CLI v0.272.0 I didn't encounter any issues.

Re: Unable to configure custom compute for DLT pipeline

GANAPATI_HEGDE — Tue, 11 Nov 2025 04:41:00 GMT

i updated my CLI and deployed the job, still i dont see the clusters updates in pipeline

Re: Unable to configure custom compute for DLT pipeline

stbjelcevic — Mon, 08 Dec 2025 21:13:58 GMT

Hi @GANAPATI_HEGDE ,

Are you still having issues with this? If so, does your bundle have a targets section? Target overrides define the runtime cluster settings that the pipeline actually uses when deployed to that target and take precedence.

Is there possibly a compute policy constraining the DLT cluster settings? If a compute policy for DLT clusters is applied (cluster_type = dlt), its constraints (like worker type and min/max workers) take precedence over your YAML cluster settings.