Hi @nayan_wylde
Yes, this is a common challenge when transitioning to Unity Catalog (UC) enabled clusters.
The installation of Maven packages from Artifactory repositories does work differently in UC environments,
but there are several approaches you can use to maintain your organization's approved patterns.
Current Limitations in UC Standard Mode:
In UC Standard mode clusters, the traditional methods of installing Maven packages directly through cluster libraries
or %pip install commands have restrictions due to the enhanced security model. This is why you're seeing success
with the manual JAR placement in volumes.
Recommended Solutions for Artifactory Integration:
1. Use Init Scripts with Artifactory Authentication
You can create an init script that downloads packages from your Artifactory during cluster startup:
#!/bin/bash
# Download from Artifactory with authentication
curl -u $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN \
-o /databricks/jars/your-package.jar \
https://your-artifactory-url/path/to/package.jar
2. Databricks Asset Bundles (Recommended)
Configure your deployment pipeline to include Artifactory packages as part of your bundle deployment.
This maintains the approved pattern while working within UC constraints.
3. Custom Package Management: Create a standardized process where:
- Packages are pulled from Artifactory during your CI/CD pipeline
- JARs are placed in designated volumes/DBFS locations
- Cluster configurations reference these standard locations.
4. Unity Catalog Volumes with Automation: Set up an automated process that:
- Periodically syncs approved packages from Artifactory to UC volumes
- Uses service principals for authentication
- Maintains version control and dependency management
LR
