JDBC with serverless compute

Dedescoat
New Contributor
Hi community,

We have a scenario where we need to ingest data into Lakebase. Currently, we are trying to use JDBC to write data in a notebook with serverless compute. However, the documentation on serverless limitations (link) mentions that JAR libraries are not supported in serverless notebooks, which prevents us from using a JDBC driver for ingestion.
 
 
Recently, a new feature JDBC connection was introduced, this allows JDBC drivers to be stored in a Unity Catalog volume and used through a configured JDBC connection. We tested this approach following the steps, and successfully wrote data into a Lakebase database from a serverless compute.
 
 
However, we noticed that the write performance is significantly slower compared to using a classic compute. We are wondering what might be causing this performance degradation. Is it an inherent limitation of serverless compute, or something specific to how JDBC connections are handled in databricks?
 

Additionally, we would like to confirm whether this is the best approach for writing data into Lakebase. An alternative we are considering is using psycopg2, but that would require manually writing sql statements.
 
Thanks, we'd appreciate any insights you might have.