One PDAL init script for every Databricks Runtime (and no apt)

DanielMcauley — Wed, 04 Nov 2026 19:00:08 GMT

I needed PDAL on Databricks for a LiDAR processing workload. PDAL is a C++ library with a long native dependency chain (GDAL, PROJ, HDF5, a recent libstdc++), and the Python bindings have to match the interpreter exactly. The usual approach, an apt-get install init script, doesn’t hold up here, for several reasons:

Each DBR major version runs on a different Ubuntu release, so package names and versions keep shifting.
Ubuntu’s PDAL is well behind upstream.
The Python bindings it installs are built for the OS Python, not the DBR’s Python.
The clusters had no public internet access at all, so apt couldn’t reach its mirrors anyway.

On top of that, the workspaces ran a mix of Dedicated and Standard (shared) clusters across several runtime versions. I wanted one script that works everywhere and doesn’t need maintaining per runtime. This is what ended up working. It’s built to run on any runtime from 13.3 up, and I verified it end to end on DBR 17.3 in both Dedicated and Standard access modes.

Stop using the OS package manager

micromamba is a single static binary with no system dependencies. The init script downloads it, then builds a self-contained environment from conda-forge:

ARCH="$(uname -m)"                        # x86_64 or aarch64
case "$ARCH" in x86_64) MAMBA_ARCH=linux-64 ;; aarch64) MAMBA_ARCH=linux-aarch64 ;; esac

curl -fsSL "https://micro.mamba.pm/api/micromamba/${MAMBA_ARCH}/latest" \
  | tar -xj -C /opt/micromamba bin/micromamba

/opt/micromamba/bin/micromamba create -y -p /opt/pdal -c conda-forge --override-channels \
  "python=${DBR_PY}" pdal python-pdal gdal proj hdf5

The environment carries its own native libraries, so the Ubuntu version underneath stops mattering, and you get current PDAL.

Match the runtime’s Python, don’t hard-code it

The key detail is python=${DBR_PY}. The script asks the runtime which Python it’s running:

DBR_PY="$(/databricks/python3/bin/python -c 'import sys;print(f"{sys.version_info.major}.{sys.version_info.minor}")')"

That’s 3.10 on DBR 13.3 and 3.12 on 17.3. conda-forge then solves for bindings that are ABI-compatible with that interpreter. That one line is why the same script runs on every runtime.

The Standard access mode surprise

On Standard mode you can hit a confusing failure: the init script runs without errors, and then import pdal fails in the notebook as if nothing had been installed.

What’s happening: on Standard (USER_ISOLATION) clusters, notebooks run as a non-root user, and after the init scripts finish, the platform re-locks root-created directories under /databricks and /tmp to 0700. The files are there, but the notebook user can’t see them, and you can’t chmod your way out of it from the init script because the re-lock happens afterwards.

/opt doesn’t get re-locked, and neither does the runtime’s base site-packages. So the environment lives in /opt/pdal, the script sets umask 022, and it explicitly makes the tree world-readable. That last step is needed because micromamba sometimes creates directories as 0700 regardless of umask.

Getting Python to find it without environment variables

On Dedicated clusters, setting PYTHONPATH and LD_LIBRARY_PATH is enough. On Standard, the notebook runs inside a sandboxed virtual environment, and I couldn’t rely on those variables making it in.

What does make it in is the runtime’s base site-packages, because those sandbox environments inherit it. So the script drops two small files there: a helper module and a one-line .pth that imports it. At interpreter startup, the helper preloads the native libraries with ctypes (so no LD_LIBRARY_PATH is needed), sets PROJ_DATA and GDAL_DATA, and appends the conda environment’s site-packages to sys.path. It appends it last, so runtime packages keep priority. If the environment isn’t there, the helper does nothing, so a bad node can’t break Python startup for everyone.

For %sh, the web terminal and Spark executors, the script also writes the usual exports to /etc/profile.d/ and spark-env.sh.

Two small things that saved support time later

The script is idempotent: if PDAL is already installed for this Python version, it skips the install.

conda-forge occasionally ships an optional PDAL plugin (the HDF reader, for example) built against a slightly different library version, and every import pdal then prints a “Can’t load library” warning. Users reasonably assume something is broken. The script now runs ldd on each plugin and removes any with missing dependencies. The core readers (LAS, LAZ, COPC, E57) aren’t affected.

Running without internet access

All downloads go through two variables:

CONDA_CHANNEL="conda-forge"                                  # or your internal conda remote
MICROMAMBA_BASE_URL="https://micro.mamba.pm/api/micromamba"  # or an internal copy

In my case both point at an internal Artifactory: a conda remote that proxies conda-forge, plus a copy of the micromamba binary. The token comes from a Databricks secret. Nothing else in the script changed.

For clusters where you can’t add an init script, I also wrote a small %run bootstrap notebook. It uses the init-script environment if it exists, and otherwise installs the same environment into node-local scratch space as the notebook user.

Installing it

Put the script in a Unity Catalog volume or workspace file, add it as a cluster-scoped init script, have an admin allowlist it for Standard mode, and do a full restart. Then check it:

%sh pdal --version
import pdal; print(pdal.info.version)

The same approach (conda-forge, the runtime’s Python version, /opt, a .pth file) has since worked for other native tools I needed on these clusters, including azcopy (see backing up Unity Catalog volumes). If you’ve been maintaining a different apt script for each runtime version, give it a try.

topic One PDAL init script for every Databricks Runtime (and no apt) in Community Articles