โ04-28-2025 06:38 AM - edited โ04-28-2025 06:40 AM
Hi everyone,
Based on the official link,
For triggered pipelines, you can select the serverless compute performance mode using the Performance optimized setting in the pipeline scheduler. When this setting is disabled, the pipeline uses standard performance mode.
we can select 2 performance modes for pipelines (i.e. delta live tables).
But I tried everything, I was not able to find the location to select/disable/enable performance modes. Could you please tell me how to do so? Or share a screenshot?
Btw, in my Azure databricks workspace, I can use serverless compute for both pipelines and jobs. I don't worry about jobs, but only pipelines (i.e. delta live tables) serverless.
Thanks in advance.
โ04-28-2025 09:34 AM
In Delta Live Tables (DLT):
You can now choose Performance Mode when scheduling a Triggered pipeline.
Two options:
Performance optimized (uses more aggressive scaling, higher resource burst)
Standard performance (normal serverless behavior)
BUT ๐ Performance mode setting only appears when you edit the pipeline schedule, not at cluster settings.
How To Select / Change Performance Mode for a DLT Pipeline
Step-by-Step (Azure Databricks):
1. Go to Workflows โ Delta Live Tables Pipelines.
2. Select your existing pipeline or Create a New Pipeline.
3. In the Pipeline Settings Page, scroll to โTrigger settingsโ or find โSchedule Triggerโ.
4. There you will find "Performance optimized" checkbox (โ
๐
If you check it โ Performance optimized mode (better scaling, faster startup, higher cost)
If you uncheck it โ Standard mode (normal scaling)
5. Save the pipeline settings and Deploy.
Continuous Pipelines do not have Performance mode option yet.
Triggered Pipelines must use serverless for this checkbox to appear.
Sometimes in older UI versions (early 2024), the checkbox might not show. Make sure your workspace is updated.
If You Donโt See the Option
Make sure your workspace is updated to DBR 13.3 LTS or later.
โ04-28-2025 01:18 PM - edited โ04-28-2025 01:20 PM
Hi @lingareddy_Alva,
Thank you so much for your reply.
I checked my existing pipelines and also tried to create a new databricks workspace (see below). But I cannot find "Trigger settings" or "Schedule Trigger" in pipeline setting. Did I miss anything?
"If You Donโt See the Option
Make sure your workspace is updated to DBR 13.3 LTS or later." Sorry, just confused here: I am using serverless dlt, I cannot configure DBR. Could you please elaborate this?
Thanks.
4 weeks ago
I would like an answer to his question also, I need to see how to turn this off, but any check box relating to performance optimization in my serverless pipeline does not show up.
3 weeks ago - last edited 3 weeks ago
Hi @BF7 @QuanSun first you select serverless and save those setting then you will see following option under job details on top right
Performance-optimized mode
Fast, performant compute startup and execution speed.
Standard mode
Disabling performance optimization gives you startup times similar to Classic infrastructure and reduces your cost.
Currently only applies to certain Serverless workloads.
Thursday
I have learned that this parameter is not governed in the pipeline configuration itself, but in the job task that runs the pipeline. This is confusing to me and I don't like it.
Passionate about hosting events and connecting people? Help us grow a vibrant local communityโsign up today to get started!
Sign Up Now