topic Re: Activate spark extensions on SQL Endpoints in Data Engineering

Activate spark extensions on SQL Endpoints

juanc — Fri, 12 Nov 2021 03:17:21 GMT

It would be possible to activate a custom extensions like Sedona (https://sedona.apache.org/download/databricks/ ) in SQL Endopoints?

Example error:

java.lang.ClassNotFoundException: org.apache.spark.sql.sedona_sql.UDT.GeometryUDT at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thrift'. (35) (SQLExecDirectW)

Re: Activate spark extensions on SQL Endpoints

Brendon_Daugher — Wed, 06 Jul 2022 20:43:54 GMT

@Kaniz Fatma does this work for SQL Endpoints as well as clusters?

I am able to install this extension using the Sedona jars with no problems on my compute cluster, but it seems to me that the UI doesn't have an option for me to do this for a SQL Warehouse. I am using azure, so I'm not sure if that makes it different.

For example, when I click into my compute cluster, I see the tab shown at the bottom, while on my "starter endpoint" I don't see an analagous option.

Thanks in advance!

Re: Activate spark extensions on SQL Endpoints

Brendon_Daugher — Wed, 06 Jul 2022 20:45:52 GMT

Re: Activate spark extensions on SQL Endpoints

Brendon_Daugher — Wed, 06 Jul 2022 20:46:04 GMT

Re: Activate spark extensions on SQL Endpoints

Arh — Mon, 20 Nov 2023 08:03:06 GMT

Having the same issue. Working with sedona in a cluster is no problem. However, when using an SQL endpoint the following error occurs:

org.apache.spark.sql.sedona_sql.UDT.GeometryUDT

Im working in Azure Databricks (latest version)

Wondering if anyone has a solutution or workaround.

Re: Activate spark extensions on SQL Endpoints

marcin-sg — Fri, 15 Dec 2023 12:39:00 GMT

Same here.

Re: Activate spark extensions on SQL Endpoints

wallfacer — Thu, 04 Apr 2024 22:29:53 GMT

@Retired_mod any updates here?

Re: Activate spark extensions on SQL Endpoints

ergü — Tue, 09 Apr 2024 08:49:28 GMT

same hier on Azure Databricks.

Re: Activate spark extensions on SQL Endpoints

naveenanto — Wed, 10 Apr 2024 16:42:45 GMT

@Retired_mod What is the right way to add custom spark extension to sql warehouse clusters?