cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to use/install a driver in Spark Declarative Pipelines (ETL)?

PabloCSD
Valued Contributor II

Salutations,

I'm using SDP for an ETL that extracts data from HANA and put it in the Unity Catalog. I defined a Policy with the needed driver:

PabloCSD_0-1768228884826.png

But I get this error:

An error occurred while calling o1013.load. : java.lang.ClassNotFoundException: com.sap.db.jdbc.Driver at java.base/

How can I install the driver?

Mahalo

4 REPLIES 4

osingh
Contributor

 

I would recommend the best ways to install the driver depending on your specific environment:

1. Recommended: Use Unity Catalog Volumes
For modern Databricks runtimes (13.3 LTS and above), storing JARs in a Unity Catalog Volume is the standard for secure, governed access.

Step 1: Create a volume and upload the ngdbc.jar file to it (you can do this via Catalog Explorer).

Step 2: When defining your UC Connection, include the path to this JAR in the java_dependencies parameter.

CREATE CONNECTION hana_connection TYPE JDBC
ENVIRONMENT (
java_dependencies '["/Volumes/your_catalog/your_schema/your_volume/ngdbc.jar"]'
)
OPTIONS (
url 'jdbc:sap://<hana_host>:<port>',
user '<user>',
password '<password>'
);

 Step 3: Ensure the cluster or job has READ VOLUME permissions on that specific volume.

2. Install as a Cluster-Scoped Library
If you are running this from a standard Job or All-Purpose cluster, you can install the driver directly to the compute resource.

  • Via Maven: This is the easiest way, as it handles dependencies automatically.
    • Go to your cluster's Libraries tab > Install New > Maven.
    • Enter the coordinates: com.sap.cloud.db.jdbc:ngdbc:2.25.9 (or your preferred version).
  • Via JAR Upload:
    • Go to Libraries > Install New > Upload > JAR.
    • Upload your local ngdbc.jar.

3. Using Databricks Asset Bundles (DABs)
If you are deploying your ETL via bundles, you must explicitly list the library in your databricks.yml task definition.

resources:
jobs:
hana_etl_job:
tasks:
- task_key: extraction_task
libraries:
- maven:
coordinates: "com.sap.cloud.db.jdbc:ngdbc:2.25.9"

Hope this should work!  

Thanks!

Om Singh

PabloCSD
Valued Contributor II

Thanks @osingh, working on that

anshu_roy
Databricks Employee
Databricks Employee

 

Hello,

It is recommended that you upload libraries to source locations that support installation onto compute with standard access mode (formerly shared access mode), as this is the recommended mode for all workloads. Please refer the documentation for best practices. Alternatively, you can also check out the JDBC unity catalog connection to read data from SAP.

PabloCSD
Valued Contributor II

I've been reading the SDP Product definition and it offers natively to load data from SAP-HANA,

PabloCSD_0-1768414499358.png

How can I access to SAP-HANA from the SDP ETL without using the driver?