Spark SQL vs serverless SQL

[camilo_s]

Are there any benchmarks showing performance and cost differences between running SQL workloads on Spark SQL vs Databricks SQL (specially serverless SQL)?

Our customer is hesitant about getting locked into Databricks SQL as opposed to being able to run their queries in Spark SQL.

Is there a performance difference between running a query on Spark SQL on a Photon-enabled cluster vs running it on serverless SQL?

[raphaelblg]

Hi @camilo_s ,

Spark SQL is the SQL API for Spark applications, while Databricks SQL is a product that follows data warehouse principles. You can anticipate performance differences mainly due to the fact that Databricks SQL compute is based on SQL Warehouses (multiple Spark clusters), while Spark SQL relies on the common Spark cluster architecture.

Choosing a serverless option may not provide huge processing time enhancements, but it can offer near-instant startup times.

I hope this information is helpful. If you have any further questions or need additional clarification, please don't hesitate to ask.

[florence023]

---

@camilo_s wrote:

Are there any benchmarks showing performance and cost differences between running SQL workloads on Spark SQL vs Databricks SQL (specially serverless SQL)?

Our customer is hesitant about getting locked into Databricks SQL as opposed to being able to run their queries in Spark SQL. hpinstantink

Is there a performance difference between running a query on Spark SQL on a Photon-enabled cluster vs running it on serverless SQL?

---

Hello,

Yes, there are benchmarks and comparisons available that highlight the performance and cost differences between running SQL workloads on Spark SQL and Databricks SQL, particularly serverless SQL.

Performance and Cost Differences:
Databricks SQL Serverless is designed to provide instant and elastic compute, which can significantly reduce costs and improve performance by eliminating the need for manual tuning1. It leverages AI to optimize performance, such as predictive I/O and automatic data layout, which can lead to substantial performance improvements1.
Spark SQL on a Photon-enabled cluster also offers performance enhancements, particularly for compute-intensive operations. Photon is a native vectorized engine that can accelerate query execution, leading to faster performance compared to traditional Spark SQL.
Benchmarks:
Internal tests by Databricks have shown that Serverless SQL can be more cost-efficient and performant compared to traditional cloud data warehouses, considering factors like cluster startup time, query execution time, and overall cost.
Comparisons using TPC-DS benchmark data indicate that Databricks SQL Serverless can outperform other platforms in terms of both execution cost and performance.
Customer Concerns:
If your customer is concerned about vendor lock-in, it’s worth noting that Databricks SQL is built on open standards and integrates well with existing Spark workloads. This means that while they can benefit from the optimizations and performance improvements of Databricks SQL, they still have the flexibility to run their queries on Spark SQL if needed.

Hope this will help you.
Best regards,
florence023