topic Re: Install system libraries on the cluster in Administration & Architecture

Install system libraries on the cluster

leungi — Fri, 10 May 2024 22:43:37 GMT

The `Library` option in cluster config allows installation of language-specific libraries - e.g., PyPi for Python, CRAN for R.

Some of these libraries - e.g., `sf` - require system libraries - e.g., `libudunits2-dev`, `libgdal-dev`.

How may one install system libraries in cluster permanently?

Re: Install system libraries on the cluster

feiyun0112 — Sat, 11 May 2024 01:21:17 GMT

you can install in init script

https://docs.databricks.com/en/init-scripts/index.html

Re: Install system libraries on the cluster

leungi — Mon, 13 May 2024 22:51:05 GMT

Thanks @feiyun0112.

For future self, steps:

1. Create a `.sh` file with the following:

#!/bin/bash set -euxo pipefail if [[ $DB_IS_DRIVER = "TRUE" ]]; then sudo apt-get -y update && apt-get install -y libudunits2-dev libgdal-dev libgeos-dev libproj-dev fi

2. Add `.sh` file to cluster by editing its settings: Advanced Options > Init Scripts.

3. Relaunch cluster.