cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

juanc
by New Contributor II
  • 2590 Views
  • 9 replies
  • 2 kudos

Activate spark extensions on SQL Endpoints

It would be possible to activate a custom extensions like Sedona (https://sedona.apache.org/download/databricks/ ) in SQL Endopoints?Example error:java.lang.ClassNotFoundException: org.apache.spark.sql.sedona_sql.UDT.GeometryUDT at org.apache.spark....

  • 2590 Views
  • 9 replies
  • 2 kudos
Latest Reply
naveenanto
New Contributor III
  • 2 kudos

@Kaniz What is the right way to add custom spark extension to sql warehouse clusters?

  • 2 kudos
8 More Replies
Mike_Gardner
by New Contributor II
  • 1144 Views
  • 2 replies
  • 3 kudos

Resolved! Data Cache in Serverless SQL Endpoints vs Non-Serverless SQL Endpoints

Do Serverless SQL Endpoints benefit from Delta and Spark Cache? If so, does it differ from a non-serverless endpoints? How long does the cache last?

  • 1144 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Mike Gardner​ , How are you doing? Was your query resolved? Do you need more help on the same?

  • 3 kudos
1 More Replies
snarfed
by New Contributor II
  • 1197 Views
  • 4 replies
  • 5 kudos

Serverless SQL endpoints on Azure?

Serverless SQL Endpoints sound exciting! Sounds like they've been in preview on AWS for a couple months. Any idea if/when they're coming to Azure?

  • 1197 Views
  • 4 replies
  • 5 kudos
Latest Reply
Kaniz
Community Manager
  • 5 kudos

Hi @snarfed​ , A SQL endpoint is a computation resource that lets you run SQL commands on data objects within Databricks SQL. This article introduces SQL endpoints and describes how to work with them using the Databricks SQL UI. A SQL endpoint is a t...

  • 5 kudos
3 More Replies
Manoj
by Contributor II
  • 6610 Views
  • 4 replies
  • 8 kudos

Resolved! Is there a way to submit multiple queries to data bricks SQL END POINT using REST API ?

Is there a way to submit multiple queries to data bricks SQL END POINT using REST API ?

  • 6610 Views
  • 4 replies
  • 8 kudos
Latest Reply
BilalAslamDbrx
Honored Contributor II
  • 8 kudos

@Manoj Kumar Rayalla​  DBSQL currently limits execution to 10 concurrent queries per cluster so there could be some queuing with 30 concurrent queries. You may want to turn on multi-cluster load balancing to horizontally scale with 1 more cluster for...

  • 8 kudos
3 More Replies
User16826992666
by Valued Contributor
  • 621 Views
  • 1 replies
  • 0 kudos

If data from a Delta table is cached in Databricks SQL and the table is altered in the backend, does it invalidate the cache?

Basically I'm worried about the scenario where data that gets cached on Databricks SQL endpoints becomes out of sync with the source Delta table. If that were to happen and data was read from the cache it would be out of date/incorrect. Is this a con...

  • 621 Views
  • 1 replies
  • 0 kudos
Latest Reply
mathan_pillai
Valued Contributor
  • 0 kudos

There are 3 types of caching. 1-Databricks SQL UI caching, 2-Query results caching , 3-Delta caching . (1) does not get invalidated. It's like your BI dashboard. BI dashboard needs to be manually refreshed.(2) and (3) gets auto invalidation.pls check...

  • 0 kudos
User16826992666
by Valued Contributor
  • 972 Views
  • 1 replies
  • 0 kudos

Resolved! When should I turn on multi-cluster load balancing on SQL Endpoints?

I see the option to enable multi-cluster load balancing when creating a SQL Endpoint, but I don't know if I should be using it or not. How do I know when I should enable it?

  • 972 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ryan_Chynoweth
Honored Contributor III
  • 0 kudos

It is best to enable multi-cluster load balance on sql endpoints when a lot of users will be running queries concurrently. Load balancing will help isolate the queries and ensure the best performance for all users. If you only have a few users runnin...

  • 0 kudos
aladda
by Honored Contributor II
  • 632 Views
  • 1 replies
  • 0 kudos
  • 632 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ryan_Chynoweth
Honored Contributor III
  • 0 kudos

Generally, interactive clusters and jobs are better suited for data engineering and transformations as they support more than just SQL. However, if you are using pure SQL, then endpoints can be used for data transformations. All of the Spark SQL fun...

  • 0 kudos
User16826994223
by Honored Contributor III
  • 375 Views
  • 0 replies
  • 0 kudos

Muti Cluster Load balancing Multi-cluster Load Balancing: the minimum and maximum number of clusters over which queries sent to the endpoint are distr...

Muti Cluster Load balancingMulti-cluster Load Balancing: the minimum and maximum number of clusters over which queries sent to the endpoint are distributed. The default is Off with a maximum of 1 cluster. When set to On, the default is minimum 1 clus...

  • 375 Views
  • 0 replies
  • 0 kudos
User16826994223
by Honored Contributor III
  • 570 Views
  • 1 replies
  • 2 kudos

Where do SQL endpoints run?

Where do Databricks SQL endpoints run?

  • 570 Views
  • 1 replies
  • 2 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 2 kudos

Like Databricks clusters, SQL endpoints are created and managed in your Cloud Account (like GCP,AZURE,cloud). SQL endpoints manage SQL-optimized clusters automatically in your account and scale to match end-user demand.

  • 2 kudos
Labels