How to attach multiple libraries to a cluster terraform in Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2021 11:59 PM
I'm currently trying to attach more than one maven artifact to my terraform configuration of a cluster.
How can we add more than one artifact in my terraform configuration ?
- Labels:
-
Maven Artifact
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-08-2021 12:00 AM
library {
maven {
coordinates = "..."
}
}
library {
maven {
coordinates = "..."
}
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-09-2022 11:57 PM
Hi @KunalGaurav,
This can be done by using a dynamic configuration block inside your databricks_cluster resource definition.
- In variable.tf make a library block as:-
variable "listOfMavenPackages" {
type = list(string)
default = [ "com.google.guava:guava:23.0" , "com.google.protobuf:protobuf-java-util:3.17.3" ]
}
- In main.tf, within databricks_cluster you may write as:-
resource "databricks_cluster" "shared_autoscaling" {
cluster_name = "Shared Autoscaling"
spark_version = data.databricks_spark_version.latest_lts.id
node_type_id = data.databricks_node_type.smallest.id
autotermination_minutes = 20
autoscale {
min_workers = 1
max_workers = 2
}
dynamic "library" {
for_each = toset(var.listOfMavenPackages)
content {
maven {
coordinates = library.value
}
}
}
}
I hope this helps you.
Simran
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-13-2023 01:13 AM
Thank you, this was exactly what I've been looking for! A way to install packages dynamically. I can confirm it works great for other methods like pypi. If any moderators see this please add this method to the documentation as it is perfect for CICD. My example below.

