topic Adding custom Jars to SQL Warehouses in Data Engineering

Adding custom Jars to SQL Warehouses

JonLaRose — Mon, 30 Oct 2023 12:21:59 GMT

Hi there,

I want to add custom JARs to an SQL warehouse (Pro if that matters) like I can in an interactive cluster, yet I don't see a way.

Is that a degraded functionality when transitioning to a SQL warehouse, or have I missed something?

Thank you.

Re: Adding custom Jars to SQL Warehouses

JonLaRose — Tue, 31 Oct 2023 11:57:38 GMT

Hi @Retired_mod ,

When trying to use the `ADD JAR` command on an SQL warehouse it returns an error saying that:

[NOT_SUPPORTED_WITH_DB_SQL] ADD JAR(S) is not supported on a SQL warehouse.

I don't think that this workaround is working.

The other option of converting JARs to Python or SQL code in not a serious answer.

Loading JARs are the standard way to load functionality to Spark SQL. Not being able to load JARs doesn't make sense in the current industry state.

Could you at least enable the `ADD JAR` command as a workaround?

Re: Adding custom Jars to SQL Warehouses

SparkJun — Fri, 31 May 2024 22:32:23 GMT

ADD JAR is a SQL syntax for Databricks runtime, it does not work for DBSQL/warehouse. DBSQL would throw this error: [NOT_SUPPORTED_WITH_DB_SQL] LIST JAR(S) is not supported on a SQL warehouse. SQLSTATE: 0A000.

This feature is not supported as of now. Please submit a feature request with Databricks Support. Thank you!