- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-18-2022 08:08 AM
Its all about spinning the spark cluster and both spark Sql api and databricks does the same operation what difference does it make to BI tools ?
- Labels:
-
Databricks SQL
-
Microstrategy
-
Spark sql
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-19-2022 12:38 AM
Hi @Basavaraj Angadi ! There's a huge difference between Spark SQL and Databricks SQL endpoints. To name a few:
- Queries are handled by an elastic load-balancer which spins up compute behind the scenes as your query load goes up or down. This load-balancer also does smart things with workload management (e.g. accelerating short queries) and caching (e.g. saving you from running a query if results are already available).
- Serverless execution. This is in preview. Unlike traditional Spark clusters, SQL endpoints can spin up compute to handle workloads much faster.
- Brand new Photon execution engine. SQL endpoints use a native C++ based execution engine which is much, much faster than the JVM-based execution engine in Spark. The best part is you don't have to change anything, your queries will just work.
- Optimized BI stack. Brand new drivers for BI and the use of cloud storage for query result retrieval mean that your BI results come back to users way faster.
In short, I would definitely recommend using SQL endpoints. They're just better for BI in every possible way!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-18-2022 12:19 PM
Hi, @Basavaraj Angadi - My name is Piper, and I'm a moderator for Databricks. Thank you for coming to the community with your question. Let's give it a while for the community to respond. We'll circle back to this if we need to.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-18-2022 11:21 PM
@Piper Wilson Thanks a lot.........Looking forward for the answers because it helps us to make a swtich from Spark SQL api to Databricks when connecting to BI tool.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-19-2022 12:38 AM
Hi @Basavaraj Angadi ! There's a huge difference between Spark SQL and Databricks SQL endpoints. To name a few:
- Queries are handled by an elastic load-balancer which spins up compute behind the scenes as your query load goes up or down. This load-balancer also does smart things with workload management (e.g. accelerating short queries) and caching (e.g. saving you from running a query if results are already available).
- Serverless execution. This is in preview. Unlike traditional Spark clusters, SQL endpoints can spin up compute to handle workloads much faster.
- Brand new Photon execution engine. SQL endpoints use a native C++ based execution engine which is much, much faster than the JVM-based execution engine in Spark. The best part is you don't have to change anything, your queries will just work.
- Optimized BI stack. Brand new drivers for BI and the use of cloud storage for query result retrieval mean that your BI results come back to users way faster.
In short, I would definitely recommend using SQL endpoints. They're just better for BI in every possible way!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-19-2022 03:38 PM
Hey Bilal,
(1) and (3) are available in databricks sql as well, right?
Results are cached on cluster and you can now spin up a photon cluster as well in high concurrency mode.
In my opinion, SQL endpoints are fast but expensive as compared to datarbricks cluster (in terms of DBU).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-01-2022 05:26 AM
@Aman Sehgal the elastic load balancer is only available in SQL endpoints.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2022 02:29 PM
Thanks @Bilal Aslam and @Aman Sehgal for jumping in!
@Basavaraj Angadi I want to make sure you got your question(s) answered! Will you let us know? Don't forget, you can select any reply as the "best answer" ! 😊

