topic Adding maven dependency to ETL pipeline in Data Engineering

Adding maven dependency to ETL pipeline

anhnnguyen — Tue, 18 Nov 2025 03:48:19 GMT

Hello guys,

I'm building ETL pipeline and need to access HANA data lake file system. In order to do that I need to have sap-hdlfs library in compute environment, library is available in maven repository.

My job will have multiple notebook task and ETL pipeline and from what I've researched, notebook tasks will use the same compute with the job, but ETL pipeline will have its own compute. And from UI, I cannot see where to add library into it.

Could anyone confirm whether my understanding is correct and how to add library to ETL pipeline compute?

Thanks in advanced.

Re: Adding maven dependency to ETL pipeline

szymon_dybczak — Tue, 18 Nov 2025 06:33:25 GMT

Hi @anhnnguyen ,

Unfortunately, Scala or Java libraries are not supported in lakeflow declarative pipeline (ETL Pipelines). So you need to use regular job if you want to install maven dependencies.

Manage Python dependencies for pipelines | Databricks on AWS

Re: Adding maven dependency to ETL pipeline

nayan_wylde — Tue, 18 Nov 2025 15:55:51 GMT

DLT doesn’t have a UI for library installation, but you can:

Use libraries configuration in the pipeline JSON or YAML spec:

{ "libraries": [ { "maven": { "coordinates": "com.sap.hana.hadoop:sap-hdlfs:<version>" } } ] }

Or, if you’re using Python, add the dependency in your requirements.txt and reference it in the pipeline settings.

Re: Adding maven dependency to ETL pipeline

anhnnguyen — Wed, 19 Nov 2025 02:03:20 GMT

I tried and it does not work. After saving config, Databricks will revert it back.

The way seem possible is load library from init script but as @szymon_dybczak mentioned, it's not a good way since it will cause unexpected behavior

Re: Adding maven dependency to ETL pipeline

XP — Wed, 19 Nov 2025 02:05:24 GMT

Hey @anhnnguyen, you can add libraries a few ways when building a notebook-based ETL pipeline:

The best practice, scalable approach to add libraries across multiple workloads or clusters is to use Policy-scoped libraries. Any compute that uses the cluster policy you define will add any dependencies to the cluster at runtime. Check this: Policy-scoped libraries

If you only need to add libraries to a single workload or cluster, use compute-scoped libraries.
Check this: Compute-scoped Libraries

Re: Adding maven dependency to ETL pipeline

anhnnguyen — Wed, 19 Nov 2025 02:30:16 GMT

thanks @XP, it worked like a charm

actually I did try with policy before but the one I tried is usage policy so that I could not find where to add library lol