Install system libraries on the cluster

[leungi]

The `Library` option in cluster config allows installation of language-specific libraries - e.g., PyPi for Python, CRAN for R.

Some of these libraries - e.g., `sf` - require system libraries - e.g., `libudunits2-dev`, `libgdal-dev`.

How may one install system libraries in cluster permanently?

[feiyun0112 · Accepted solution]

you can install in init script

https://docs.databricks.com/en/init-scripts/index.html 

[leungi]

Thanks @feiyun0112.

For future self, steps:

1. Create a `.sh` file with the following:

#!/bin/bash

set -euxo pipefail

if [[ $DB_IS_DRIVER = "TRUE" ]]; then
  sudo apt-get -y update && apt-get install -y libudunits2-dev libgdal-dev libgeos-dev libproj-dev
fi

2. Add `.sh` file to cluster by editing its settings: Advanced Options > Init Scripts.

3. Relaunch cluster.