topic Re: Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point? in Data Engineering

Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

BasavarajAngadi — Fri, 18 Feb 2022 16:08:39 GMT

Its all about spinning the spark cluster and both spark Sql api and databricks does the same operation what difference does it make to BI tools ?

Re: Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

Anonymous — Fri, 18 Feb 2022 20:19:02 GMT

Hi, @Basavaraj Angadi - My name is Piper, and I'm a moderator for Databricks. Thank you for coming to the community with your question. Let's give it a while for the community to respond. We'll circle back to this if we need to.

Re: Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

BasavarajAngadi — Sat, 19 Feb 2022 07:21:24 GMT

@Piper Wilson Thanks a lot.........Looking forward for the answers because it helps us to make a swtich from Spark SQL api to Databricks when connecting to BI tool.

Re: Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

BilalAslamDbrx — Sat, 19 Feb 2022 08:38:06 GMT

Hi @Basavaraj Angadi ! There's a huge difference between Spark SQL and Databricks SQL endpoints. To name a few:

Queries are handled by an elastic load-balancer which spins up compute behind the scenes as your query load goes up or down. This load-balancer also does smart things with workload management (e.g. accelerating short queries) and caching (e.g. saving you from running a query if results are already available).
Serverless execution. This is in preview. Unlike traditional Spark clusters, SQL endpoints can spin up compute to handle workloads much faster.
Brand new Photon execution engine. SQL endpoints use a native C++ based execution engine which is much, much faster than the JVM-based execution engine in Spark. The best part is you don't have to change anything, your queries will just work.
Optimized BI stack. Brand new drivers for BI and the use of cloud storage for query result retrieval mean that your BI results come back to users way faster.

In short, I would definitely recommend using SQL endpoints. They're just better for BI in every possible way!

Re: Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

AmanSehgal — Sat, 19 Feb 2022 23:38:46 GMT

Hey Bilal,

(1) and (3) are available in databricks sql as well, right?

Results are cached on cluster and you can now spin up a photon cluster as well in high concurrency mode.

In my opinion, SQL endpoints are fast but expensive as compared to datarbricks cluster (in terms of DBU).

Re: Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

Anonymous — Tue, 22 Feb 2022 22:29:00 GMT

Thanks @Bilal Aslam and @Aman Sehgal for jumping in!

@Basavaraj Angadi I want to make sure you got your question(s) answered! Will you let us know? Don't forget, you can select any reply as the "best answer" ! 😊

Re: Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

BilalAslamDbrx — Tue, 01 Mar 2022 13:26:14 GMT

@Aman Sehgal the elastic load balancer is only available in SQL endpoints.