Louis_Frolio
Databricks Employee
Databricks Employee

‎04-08-2025 07:04 AM

Azure Databricks Serverless Compute does support Maven library installations; however, there are some important details and limitations to consider:
  1. Support for Maven Libraries:
    • You can install libraries from a public Maven repository when working with serverless compute. This can include specifying Maven coordinates like { "coordinates": "org.jsoup:jsoup:1.7.2" }.
    • Custom Maven repositories are also supported, provided they allow unauthenticated access. Libraries are resolved in the Databricks Control Plane, meaning the repository must be accessible from there.
  2. Limitations:
    • While Maven library installation is supported, Serverless Compute introduces certain constraints. For instance:
      • Classic Databricks features such as init scripts and cluster-scoped libraries are not supported. Instead, Databricks recommends leveraging environments for dependency management in Serverless Compute.
      • Libraries installed may not persist across multiple executions, as serverless compute environments are optimized for fast start-up and may be stateless.
    • Other features, such as Spark-level observability (e.g., accessing Spark UI, detailed driver/executor logs) or tight control over Spark configurations, might also be limited.
  3. Recommendations:
    • Use environments to cache dependencies where possible, as this reduces the need to reinstall libraries with every execution.
    • Specify Maven repositories clearly if using custom ones, ensuring they are accessible from the Databricks Control Plane.
    • Test libraries in a similar environment before deploying them in serverless compute to ensure compatibility.
0 Kudos