I am trying to create a cluster configuration using DABS and defining library dependencies.
My yaml file looks like this:
resources:
clusters:
project_Job_Cluster:
cluster_name: "Project Cluster"
spark_version: "16.3.x-cpu-ml-scala2.12"
node_type_id: "Standard_DS3_v2"
#num_workers: 0
autotermination_minutes: 20
spark_conf:
"spark.databricks.delta.preview.enabled": "true"
"spark.databricks.cluster.profile": "singleNode"
"spark.master": "local[*, 4]"
custom_tags:
"Project": "Project"
libraries:
- pypi:
package: "databricks-sql-connector"
- pypi:
package: "python-docx"
The cluster configuration is recognised and it creates it but without the specified libraries and I get a warning:
"Warning: unknown field: libraries at resources.clusters.project_Job_Cluster in resources/project job cluster.yml:16:7"
Which leads me to conclude the I cannot use "libraries" in cluster config?What is the right way?