I needed PDAL on Databricks for a LiDAR processing workload. PDAL is a C++ library with a long native dependency chain (GDAL, PROJ, HDF5, a recent libstdc++), and the Python bindings have to match the interpreter exactly. The usual approach, an apt-get install init script, doesnโt hold up here, for several reasons:
- Each DBR major version runs on a different Ubuntu release, so package names and versions keep shifting.
- Ubuntuโs PDAL is well behind upstream.
- The Python bindings it installs are built for the OS Python, not the DBRโs Python.
- The clusters had no public internet access at all, so apt couldnโt reach its mirrors anyway.
On top of that, the workspaces ran a mix of Dedicated and Standard (shared) clusters across several runtime versions. I wanted one script that works everywhere and doesnโt need maintaining per runtime. This is what ended up working. Itโs built to run on any runtime from 13.3 up, and I verified it end to end on DBR 17.3 in both Dedicated and Standard access modes.
Stop using the OS package manager
micromamba is a single static binary with no system dependencies. The init script downloads it, then builds a self-contained environment from conda-forge:
ARCH="$(uname -m)" # x86_64 or aarch64
case "$ARCH" in x86_64) MAMBA_ARCH=linux-64 ;; aarch64) MAMBA_ARCH=linux-aarch64 ;; esac
curl -fsSL "https://micro.mamba.pm/api/micromamba/${MAMBA_ARCH}/latest" \
| tar -xj -C /opt/micromamba bin/micromamba
/opt/micromamba/bin/micromamba create -y -p /opt/pdal -c conda-forge --override-channels \
"python=${DBR_PY}" pdal python-pdal gdal proj hdf5
The environment carries its own native libraries, so the Ubuntu version underneath stops mattering, and you get current PDAL.
Match the runtimeโs Python, donโt hard-code it
The key detail is python=${DBR_PY}. The script asks the runtime which Python itโs running:
DBR_PY="$(/databricks/python3/bin/python -c 'import sys;print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
Thatโs 3.10 on DBR 13.3 and 3.12 on 17.3. conda-forge then solves for bindings that are ABI-compatible with that interpreter. That one line is why the same script runs on every runtime.
The Standard access mode surprise
On Standard mode you can hit a confusing failure: the init script runs without errors, and then import pdal fails in the notebook as if nothing had been installed.
Whatโs happening: on Standard (USER_ISOLATION) clusters, notebooks run as a non-root user, and after the init scripts finish, the platform re-locks root-created directories under /databricks and /tmp to 0700. The files are there, but the notebook user canโt see them, and you canโt chmod your way out of it from the init script because the re-lock happens afterwards.
/opt doesnโt get re-locked, and neither does the runtimeโs base site-packages. So the environment lives in /opt/pdal, the script sets umask 022, and it explicitly makes the tree world-readable. That last step is needed because micromamba sometimes creates directories as 0700 regardless of umask.
Getting Python to find it without environment variables
On Dedicated clusters, setting PYTHONPATH and LD_LIBRARY_PATH is enough. On Standard, the notebook runs inside a sandboxed virtual environment, and I couldnโt rely on those variables making it in.
What does make it in is the runtimeโs base site-packages, because those sandbox environments inherit it. So the script drops two small files there: a helper module and a one-line .pth that imports it. At interpreter startup, the helper preloads the native libraries with ctypes (so no LD_LIBRARY_PATH is needed), sets PROJ_DATA and GDAL_DATA, and appends the conda environmentโs site-packages to sys.path. It appends it last, so runtime packages keep priority. If the environment isnโt there, the helper does nothing, so a bad node canโt break Python startup for everyone.
For %sh, the web terminal and Spark executors, the script also writes the usual exports to /etc/profile.d/ and spark-env.sh.
Two small things that saved support time later
The script is idempotent: if PDAL is already installed for this Python version, it skips the install.
conda-forge occasionally ships an optional PDAL plugin (the HDF reader, for example) built against a slightly different library version, and every import pdal then prints a โCanโt load libraryโ warning. Users reasonably assume something is broken. The script now runs ldd on each plugin and removes any with missing dependencies. The core readers (LAS, LAZ, COPC, E57) arenโt affected.
Running without internet access
All downloads go through two variables:
CONDA_CHANNEL="conda-forge" # or your internal conda remote
MICROMAMBA_BASE_URL="https://micro.mamba.pm/api/micromamba" # or an internal copy
In my case both point at an internal Artifactory: a conda remote that proxies conda-forge, plus a copy of the micromamba binary. The token comes from a Databricks secret. Nothing else in the script changed.
For clusters where you canโt add an init script, I also wrote a small %run bootstrap notebook. It uses the init-script environment if it exists, and otherwise installs the same environment into node-local scratch space as the notebook user.
Installing it
Put the script in a Unity Catalog volume or workspace file, add it as a cluster-scoped init script, have an admin allowlist it for Standard mode, and do a full restart. Then check it:
%sh pdal --version
import pdal; print(pdal.info.version)
The same approach (conda-forge, the runtimeโs Python version, /opt, a .pth file) has since worked for other native tools I needed on these clusters, including azcopy (see backing up Unity Catalog volumes). If youโve been maintaining a different apt script for each runtime version, give it a try.
