Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

[BasavarajAngadi]

Its all about spinning the spark cluster and both spark Sql api and databricks does the same operation what difference does it make to BI tools ?

[Anonymous]

Hi, @Basavaraj Angadi​ - My name is Piper, and I'm a moderator for Databricks. Thank you for coming to the community with your question. Let's give it a while for the community to respond. We'll circle back to this if we need to.

[BasavarajAngadi]

@Piper Wilson​  Thanks a lot.........Looking forward for the answers because it helps us to make a swtich from Spark SQL api to Databricks when connecting to BI tool.

[BilalAslamDbrx · Accepted solution]

Hi @Basavaraj Angadi​ ! There's a huge difference between Spark SQL and Databricks SQL endpoints. To name a few:

• Queries are handled by an elastic load-balancer which spins up compute behind the scenes as your query load goes up or down. This load-balancer also does smart things with workload management (e.g. accelerating short queries) and caching (e.g. saving you from running a query if results are already available).
• Serverless execution. This is in preview. Unlike traditional Spark clusters, SQL endpoints can spin up compute to handle workloads much faster.
• Brand new Photon execution engine. SQL endpoints use a native C++ based execution engine which is much, much faster than the JVM-based execution engine in Spark. The best part is you don't have to change anything, your queries will just work.
• Optimized BI stack. Brand new drivers for BI and the use of cloud storage for query result retrieval mean that your BI results come back to users way faster.

In short, I would definitely recommend using SQL endpoints. They're just better for BI in every possible way!

[AmanSehgal]

Hey Bilal,

(1) and (3) are available in databricks sql as well, right?

Results are cached on cluster and you can now spin up a photon cluster as well in high concurrency mode.

In my opinion, SQL endpoints are fast but expensive as compared to datarbricks cluster (in terms of DBU).

[BilalAslamDbrx]

@Aman Sehgal​ the elastic load balancer is only available in SQL endpoints.

[Anonymous]

Thanks @Bilal Aslam​ and @Aman Sehgal​ for jumping in!

@Basavaraj Angadi​ ​ I want to make sure you got your question(s) answered! Will you let us know? Don't forget, you can select any reply as the "best answer" ! 😊