topic Re: Problem with Databricks JDBC connection: Error occured while deserializing arrow data in Data Engineering

Problem with Databricks JDBC connection: Error occured while deserializing arrow data

tz1 — Wed, 02 Mar 2022 16:14:02 GMT

I have a Java program like this to test out the Databricks JDBC connection with the Databricks JDBC driver.

        Connection connection = null;
        try {
            Class.forName(driver);
            connection = DriverManager.getConnection(url, username, password);
            if (connection != null) {
                System.out.println("Connection Established");
            } else {
                System.out.println("Connection Failed");
            }
            Statement statement = connection.createStatement();
            ResultSet rs = statement.executeQuery("select * from standard_info_service.daily_transactions"); 
            while (rs.next()) {
                System.out.print("created_date: " + rs.getInt("created_date") + ", ");
                System.out.println("daily_transactions: " + rs.getInt("daily_transactions"));
            }
        } catch (Exception e) {
            System.out.println(e);
        }

This program, however, throws an error like this:

Connection Established
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
java.sql.SQLException: [Simba][SparkJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

What will be the solution?

Re: Problem with Databricks JDBC connection: Error occured while deserializing arrow data

tz1 — Thu, 03 Mar 2022 19:19:47 GMT

Thanks @Kaniz Fatma

openjdk-17.0.2

Databricks JDBC Driver 2.6.22

Re: Problem with Databricks JDBC connection: Error occured while deserializing arrow data

Alice__Caterpil — Thu, 10 Mar 2022 02:09:48 GMT

This error is mentioned in Spark documentation - https://spark.apache.org/docs/latest/, looks like this is specific to the version of Java and can be avoid by having the mentioned properties set

Re: Problem with Databricks JDBC connection: Error occured while deserializing arrow data

jose_gonzalez — Tue, 22 Mar 2022 16:50:46 GMT

Hi @Tony Zhou ,

Just a friendly follow-up. Did @Kaniz Fatma 's response helped you to resolve this issue? if not, please share more details, like the full error stack trace and some code snippets.

Re: Problem with Databricks JDBC connection: Error occured while deserializing arrow data

Alice__Caterpil — Thu, 24 Mar 2022 00:19:51 GMT

Hi @Jose Gonzalez ,

This similar issue in snowflake in JDBC is a good reference, I was able to get this to work in Java OpenJDK 17 by having this JVM option specified:

--add-opens=java.base/java.nio=ALL-UNNAMED

Although I came across another issue with using apache DHCP to connect to Databricks SQL endpoint:

Caused by: java.sql.SQLFeatureNotSupportedException: [Simba][JDBC](10220) Driver does not support this optional feature.

at com.simba.spark.exceptions.ExceptionConverter.toSQLException(Unknown Source)

at com.simba.spark.jdbc.common.SConnection.setAutoCommit(Unknown Source)

at com.simba.spark.jdbc.jdbc42.DSS42Connection.setAutoCommit(Unknown Source)

at org.apache.commons.dbcp2.DelegatingConnection.setAutoCommit(DelegatingConnection.java:801)

The same problem occurred after I switched to Hikari.

Finally, I got it working by just using Basic DataSource and set auto-commit to False. BasicDataSource is not suitable for production though, would there be a new driver release that can handle this better?

Re: Problem with Databricks JDBC connection: Error occured while deserializing arrow data

AmeyJoshi — Thu, 24 Mar 2022 15:58:30 GMT

Thanks a lot @Alice Hung your suggestion works. I am really grateful to you for sharing it. There is absolutely no help available elsewhere.

Re: Problem with Databricks JDBC connection: Error occured while deserializing arrow data

AmeyJoshi — Thu, 24 Mar 2022 16:47:14 GMT

Definitely, I would want to. But I can't find an option to mark it as the best.

Re: Problem with Databricks JDBC connection: Error occured while deserializing arrow data

AmeyJoshi — Thu, 24 Mar 2022 17:07:35 GMT

I am sorry @Kaniz Fatma but I don't see that option available to me. If you see it, kindly use it on my behalf.