topic Re: Adding JAR from Azure DevOps Artifacts feed to Databricks job in Data Engineering

Adding JAR from Azure DevOps Artifacts feed to Databricks job

yannickmo — Tue, 12 Oct 2021 15:01:59 GMT

Hello,

We have some Scala code which is compiled and published to an Azure DevOps Artifacts feed.

The issue is we're trying to now add this JAR to a Databricks job (through Terraform) to automate the creation.

To do this I'm trying to authenticate using a generated token but am now getting an error when trying to add the library:

Run result unavailable: job failed with error message
 Library installation failed for library due to user error for maven {
  coordinates: "groupId:artifactId:version"
  repo: "https://user:token@ORG.pkgs.visualstudio.com/FEED_ID/_packaging/FEED_NAME/maven/v1"
}

Any idea how to set this up properly? I know people have figured this out for PyPi but haven't found anything for Maven.

The other option is putting it on ADLS and adding it from there, but I'd prefer direct integration without a middle-man.

Re: Adding JAR from Azure DevOps Artifacts feed to Databricks job

Hubert-Dudek — Tue, 12 Oct 2021 15:45:51 GMT

In past it was said that databrics doesn't support internal maven libraries. Now I don't see anymore that words in documentation but I guess is still like that.

only idea which I have is to use Azure Pipelines +

databricks libraries install --cluster-id 1234-567890-lest123 --jar dbfs:/test-dir/test.jar

Re: Adding JAR from Azure DevOps Artifacts feed to Databricks job

yannickmo — Wed, 13 Oct 2021 07:14:55 GMT

Hey thanks for the reply!

Yeah I read the same about internal maven libraries, but I'm not sure what "internal" means in this context. If I try to resolve the repository URL using user:token locally it authenticates perfectly fine from a public address.

The problem is that the error given by Databricks is the same even when you remove the user:token from the URL so you don't get any feedback what's going wrong.

Your idea is indeed also a possible solution but it's similar to putting it on ADLS and installing from there.

Re: Adding JAR from Azure DevOps Artifacts feed to Databricks job

yannickmo — Wed, 13 Oct 2021 07:17:30 GMT

Hey thanks for the welcome and likewise!

I do get that using Scala isn't as popular as Python for Databricks and using JARs to assemble jobs makes it even more special of a use-case, but if this is possible it would align nicely for automation.

Re: Adding JAR from Azure DevOps Artifacts feed to Databricks job

yannickmo — Mon, 18 Oct 2021 12:58:51 GMT

Any other ideas?

Re: Adding JAR from Azure DevOps Artifacts feed to Databricks job

alexott — Thu, 25 Nov 2021 18:47:59 GMT

As of right now, Databricks can't use non-public Maven repositories as resolving of the maven coordinates happens in the control plane. That's different from the R & Python libraries. As workaround you may try to install libraries via init script or upload to ADLS or S3, and refer by URL from the cluster config

Re: Adding JAR from Azure DevOps Artifacts feed to Databricks job

BobGeor_68322 — Tue, 16 Aug 2022 15:55:07 GMT

@Alex Ott do you have an example of an init script that can copy the jar file from ADLS to /databricks/jars? I cannot seem to get connected to ADLS over abfss nor https from an init script.

Re: Adding JAR from Azure DevOps Artifacts feed to Databricks job

alexott — Fri, 19 Aug 2022 12:28:03 GMT

It's two different things:

installing via init script - in it you can just do

cp /dbfs/FileStore/jars/name1.jar /databricks/jars/

Installing library from ADLS - just specify ADLS URL in the cluster UI, and make sure that cluster has service principal attached so the file could be downloaded