05-06-2022 03:25 AM
05-12-2022 11:20 PM
Great question! There are similarities and differences:
Similarities
Differences
TL;DR if you are doing SQL/BI, please consider using SQL Endpoints, it's generally the best choice for that workload.
05-06-2022 04:40 AM
They are very similar. Databricks SQL uses compute that has photon enabled. A traditional cluster with photon enabled does allow for a few more configurations to be set around the cluster architecture and settings. The traditional cluster will also have more libraries installed as it needs to run things in various languages, where the endpoints only needs SQL APIs.
https://docs.databricks.com/runtime/photon.html#limitations. This lists some limitations, although additional data source reads is in preview now.
05-06-2022 06:08 AM
Thank you. Will traditional cluster support serverless execution in the future or only SQL endpoints support that?
And are there any optimization tweaks in Databricks SQL that makes it perhaps faster than traditional Databricks cluster running only SQL queries?
05-06-2022 08:06 AM
Serverless for traditional compute is in preview for single node machines and multinode cluster serverless is on the roadmap.
I'm sure there are a few optimizations that makes things faster. Simple things such as caching metadata in the metastore helps.
05-06-2022 08:21 AM
I wouldn't call them the same as Databricks SQL runtime is a bit different (not everything is supported for example UDFs), new releases are separated from standard runtimes updates: https://docs.databricks.com/sql/release-notes/index.html
Databricks cluster can handle notebooks. SQL endpoint is only for SQL queries.
Both can be in photon or non-photon versions. Photon has a bunch of improvements for example better handle small files problem.
05-12-2022 11:20 PM
Great question! There are similarities and differences:
Similarities
Differences
TL;DR if you are doing SQL/BI, please consider using SQL Endpoints, it's generally the best choice for that workload.
05-13-2022 03:56 AM
Hi @John William, Just a friendly follow-up. Do you still need help or the above responses help you to find the solution? Please let us know.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.