cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to select performance mode for Databricks Delta Live Tables

QuanSun
New Contributor II

Hi everyone,

Based on the official link,

For triggered pipelines, you can select the serverless compute performance mode using the Performance optimized setting in the pipeline scheduler. When this setting is disabled, the pipeline uses standard performance mode.

we can select 2 performance modes for pipelines (i.e. delta live tables).

But I tried everything, I was not able to find the location to select/disable/enable performance modes. Could you please tell me how to do so? Or share a screenshot?

Btw, in my Azure databricks workspace, I can use serverless compute for both pipelines and jobs. I don't worry about jobs, but only pipelines (i.e. delta live tables) serverless.

Thanks in advance.

2 REPLIES 2

LRALVA
Honored Contributor

@QuanSun 

In Delta Live Tables (DLT):
You can now choose Performance Mode when scheduling a Triggered pipeline.
Two options:
Performance optimized (uses more aggressive scaling, higher resource burst)
Standard performance (normal serverless behavior)
BUT ๐Ÿ‘‰ Performance mode setting only appears when you edit the pipeline schedule, not at cluster settings.

How To Select / Change Performance Mode for a DLT Pipeline
Step-by-Step (Azure Databricks):
1. Go to Workflows โ†’ Delta Live Tables Pipelines.
2. Select your existing pipeline or Create a New Pipeline.
3. In the Pipeline Settings Page, scroll to โ€œTrigger settingsโ€ or find โ€œSchedule Triggerโ€.
4. There you will find "Performance optimized" checkbox (โœ…๐Ÿ˜ž
If you check it โ†’ Performance optimized mode (better scaling, faster startup, higher cost)
If you uncheck it โ†’ Standard mode (normal scaling)
5. Save the pipeline settings and Deploy.

 

Continuous Pipelines do not have Performance mode option yet.
Triggered Pipelines must use serverless for this checkbox to appear.
Sometimes in older UI versions (early 2024), the checkbox might not show. Make sure your workspace is updated.
If You Donโ€™t See the Option
Make sure your workspace is updated to DBR 13.3 LTS or later.

LR

QuanSun
New Contributor II

Hi @LRALVA,

Thank you so much for your reply.

I checked my existing pipelines and also tried to create a new databricks workspace (see below). But I cannot find "Trigger settings" or "Schedule Trigger" in pipeline setting. Did I miss anything?

image.png

"If You Donโ€™t See the Option
Make sure your workspace is updated to DBR 13.3 LTS or later." Sorry, just confused here: I am using serverless dlt, I cannot configure DBR. Could you please elaborate this?

Thanks.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now