cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Activate spark extensions on SQL Endpoints

juanc
New Contributor II

It would be possible to activate a custom extensions like Sedona (https://sedona.apache.org/download/databricks/ ) in SQL Endopoints?

Example error:

java.lang.ClassNotFoundException: org.apache.spark.sql.sedona_sql.UDT.GeometryUDT at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thrift'. (35) (SQLExecDirectW)

8 REPLIES 8

Brendon_Daugher
New Contributor II

@Kaniz Fatma​ does this work for SQL Endpoints as well as clusters?

I am able to install this extension using the Sedona jars with no problems on my compute cluster, but it seems to me that the UI doesn't have an option for me to do this for a SQL Warehouse. I am using azure, so I'm not sure if that makes it different.

For example, when I click into my compute cluster, I see the tab shown at the bottom, while on my "starter endpoint" I don't see an analagous option.

Thanks in advance!

Brendon_Daugher
New Contributor II
 

Brendon_Daugher
New Contributor II
 

Arh
New Contributor II

Having the same issue. Working with sedona in a cluster is no problem. However, when using an SQL endpoint the following error occurs:

org.apache.spark.sql.sedona_sql.UDT.GeometryUDT

Im working in Azure Databricks (latest version)

 

Wondering if anyone has a solutution or workaround.

marcin-sg
New Contributor III

Same here.

wallfacer
New Contributor II

@Retired_mod any updates here?

ergü
New Contributor II

same hier on Azure Databricks.

naveenanto
New Contributor III

@Retired_mod What is the right way to add custom spark extension to sql warehouse clusters?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group