Serverless costs, standard vs performance optimized

[rcostanza]

In the pricing page for Lakeflow Declarative Pipelines (formerly DLT), for serverless it shows a single cost of $0.35/DBU for both standard and performance optimized. But in the feature table below, it says standard is "Up to 70% cheaper than running in the Performance optimized mode".

I'm trying to estimate costs and I'm not sure how, since I don't know which mode runs for the listed cost. Is standard cheaper or performance costlier? How do I compare costs between the two?

[Isi]

Hello @rcostanza !

While the DBU price is the same for both modes, the Performance Optimized mode behaves quite differently:

Performance Optimized Mode prioritizes speed and throughput. It aggressively scales resources to minimize processing time, which can significantly increase total cost due to higher DBU consumption.

The actual price you pay will depend on your workload. Databricks manages the underlying infrastructure using real-time workload intelligence and automatically scales clusters as needed. Its enhanced autoscaler will also wind down resources when it detects they’re no longer required. You can find the DBU usage of serverless Lakeflow Declarative Pipelines by querying the billable usage table, part of the Databricks system tables

"DBUs are a measure of the processing power used to run your workload. Databricks manages the underlying infrastructure and uses real-time intelligence about your workload to determine how to efficiently scale resources to meet its needs. The best way to estimate serverless DBUs is to run a few representative workloads on serverless and measure the resultant DBUs."

Also note to review your use case:

"Standard Performance mode is only available for triggered pipelines. Continuous execution is not supported."

Hope this helps, 🙂

Isi