- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-18-2022 08:08 AM
Its all about spinning the spark cluster and both spark Sql api and databricks does the same operation what difference does it make to BI tools ?
- Labels:
-
Databricks SQL
-
Microstrategy
-
Spark sql
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-19-2022 12:38 AM
Hi @Basavaraj Angadiโ ! There's a huge difference between Spark SQL and Databricks SQL endpoints. To name a few:
- Queries are handled by an elastic load-balancer which spins up compute behind the scenes as your query load goes up or down. This load-balancer also does smart things with workload management (e.g. accelerating short queries) and caching (e.g. saving you from running a query if results are already available).
- Serverless execution. This is in preview. Unlike traditional Spark clusters, SQL endpoints can spin up compute to handle workloads much faster.
- Brand new Photon execution engine. SQL endpoints use a native C++ based execution engine which is much, much faster than the JVM-based execution engine in Spark. The best part is you don't have to change anything, your queries will just work.
- Optimized BI stack. Brand new drivers for BI and the use of cloud storage for query result retrieval mean that your BI results come back to users way faster.
In short, I would definitely recommend using SQL endpoints. They're just better for BI in every possible way!

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-18-2022 12:19 PM
Hi, @Basavaraj Angadiโ - My name is Piper, and I'm a moderator for Databricks. Thank you for coming to the community with your question. Let's give it a while for the community to respond. We'll circle back to this if we need to.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-18-2022 11:21 PM
@Piper Wilsonโ Thanks a lot.........Looking forward for the answers because it helps us to make a swtich from Spark SQL api to Databricks when connecting to BI tool.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-19-2022 12:38 AM
Hi @Basavaraj Angadiโ ! There's a huge difference between Spark SQL and Databricks SQL endpoints. To name a few:
- Queries are handled by an elastic load-balancer which spins up compute behind the scenes as your query load goes up or down. This load-balancer also does smart things with workload management (e.g. accelerating short queries) and caching (e.g. saving you from running a query if results are already available).
- Serverless execution. This is in preview. Unlike traditional Spark clusters, SQL endpoints can spin up compute to handle workloads much faster.
- Brand new Photon execution engine. SQL endpoints use a native C++ based execution engine which is much, much faster than the JVM-based execution engine in Spark. The best part is you don't have to change anything, your queries will just work.
- Optimized BI stack. Brand new drivers for BI and the use of cloud storage for query result retrieval mean that your BI results come back to users way faster.
In short, I would definitely recommend using SQL endpoints. They're just better for BI in every possible way!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-19-2022 03:38 PM
Hey Bilal,
(1) and (3) are available in databricks sql as well, right?
Results are cached on cluster and you can now spin up a photon cluster as well in high concurrency mode.
In my opinion, SQL endpoints are fast but expensive as compared to datarbricks cluster (in terms of DBU).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ03-01-2022 05:26 AM
@Aman Sehgalโ the elastic load balancer is only available in SQL endpoints.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-22-2022 02:29 PM
Thanks @Bilal Aslamโ and @Aman Sehgalโ for jumping in!
@Basavaraj Angadiโ โ I want to make sure you got your question(s) answered! Will you let us know? Don't forget, you can select any reply as the "best answer" ! ๐

